Skip to end of metadata
Go to start of metadata

On 10/26 we will be taking our globus server offline for upgrade to the latest version of globus.  We estimate the downtime to last less than two hours.  Please be sure to time transfers accordingly, ongoing transfers will fail.


8:42am We have completed the switch to our new Globus server.  Currently guest collections are not working we have a ticket open with Globus support for this.


3pm:  All globus services should now be available.

We will be updating  our RDP gateway software to the latest revision on 11/2.  The server will be offline for a short time during this upgrade.  We anticipate the server being offline for less than two hours.

Tuesday Oct 13 at 10:00am dev-amd20 will be shutdown for maintenance.  The outage should be brief and the system will be returned to service as quickly as possible.

At 8:00PM on Saturday, October 10th, the SLURM scheduler will be going offline for maintenance. Client commands will not be available during this time, e.g. sbatch, squeue, srun. Running jobs will not be affected. Maintenance is expected to last less than one hour. If you have any questions, please contact us https://contact.icer.msu.edu/.

Unexpected Lustre Outage

On Sunday, October 4th, from approximately 9:00 AM to 10:20 AM, the Luster file system was hung after it's metadata server ran out of disk space. Additional space was added and functionality restored. Jobs using the Lustre file system during this time may have experienced I/O errors.

General availability for the AMD20 cluster (ICER’s new supercomputer) began at 9 AM on Tuesday, September 29th. Please report any issues that you see to ICER through the help ticket system.

We also re-enabled the automatic 45 day purge on /mnt/ls15 on October 1st.

The first wave of AMD20 (210 CPU nodes and 9 GPU nodes) is now available for testing in the amd20-test partition. 

Use:

#SBATCH -q amd20-test
#SBATCH -p amd20-test

to request the test partition and QOS.

The dev-amd20 and dev-amd20-v100 nodes are available from other development nodes.

There is no limit on cores you can use, but a 24 hour limit on CPU time. Systems may need to be restarted at any time as we complete testing and address any issues that may arise.

If everything goes well we anticipate that this system will be available within the normal scheduler by the end of the month.

For more information, please see:

Cluster amd20 with AMD CPUs

Please contact us if you notice any issues or have additional questions.


9-2-20 at 12am we will be taking the globus google drive server offline to maintenance.  We will be attempting to correct an issue that is causing only the My Drive space to be available.  The maintenance is expected to last up to 4 hours.  When the server is back online users may need to remove their old collections and remap a new collection.



Update 08/13/20 1:00PM: A patch has been applied and scheduler functionality has returned to normal 

    We are currently encountering some performance issues with the job scheduler following updates during the maintenance. This is causing jobs not to schedule properly as well as delays in job execution. We are working to resolve this with our vendor.

HPCC provides 2 nodes of the new purchased cluster amd20 for users to do testing. Please check the wiki page "Running Job on amd20 Test Nodes" for how to run your jobs on the nodes. Users can also find more information (such as node performance, AMD Libraries) about the cluster through the page "Cluster amd20 with AMD CPU".

We have firewall issue after the HPCC maintenance (on August 4th). Sometimes the network situation is good but sometimes it is very slow. If you log into HPCC gateway and get the response like "Last login:  ... ...", please wait for further responses which might take a while. After the waiting, you are logged in. Our system administrator is now working with ITS to resolve this issue.

We are currently experiencing high CPU load on ICER's firewall. Users may experience lag when accessing files using the gateway nodes; users are advised to use development nodes until we resolve the issue. MSU IT Security is working with the firewall vendor to diagnose and resolve the issue.

The HPCC will be unavailable on August 4th, 2020 to do regularly scheduled software, hardware, and network maintenance and to prepare for the new cluster installation. During the maintenance window, interactive access via SSH and OpenOnDemand will be disabled, remote home directory access (via Globus and Windows File Sharing) will be blocked, and no jobs that would overlap the maintenance window will be started until after it completes. Please contact us if you have any questions or concerns.

Update 1 AM 08-04: All services are currently unavailable; initial software updates have been staged and the network equipment is being updated.

Update 3 AM 08-04: The core network upgrades are complete.

Update 10 AM 08-04: Scheduler updates are complete. Compute node updates are underway. Windows file sharing access to the home directory servers are available.

Update 4 PM 08-04: Compute node updates are nearly complete, we anticipate a return to service by 5 PM today. There is an issue with one of our license servers; some licensed software may fail when started. We are working with the vendors to update the configuration.

Update 6:30 PM 08-04: Interactive access has been resumed. Late in the process we experienced a component failure on the 2016 cluster that has delayed our return to scheduling. We have restored some of the licenses on the failed server and are working with vendors to move the rest to a new license server.

Update 8:00 PM 08-04: The scheduler has been resumed and we have returned to full service.  We're finishing up a few outstanding issues; if you have any issues please contact us.

Update: 12:00 PM 08-12: The license sever issue is resolved.

During our last upgrade for legacy scratch upgrade in December of last year the normal purge of files not modified in 45 days became disabled.  We will be enabling this purge again during the next outage.  Please make sure to check your legacy scratch directories and backup any data that you may need.


The HPCC's main scratch system (gs18) is nearing capacity. We ask that users reduce their usage or move work to ls15. If gs18 remains near capacity, a more aggressive purge policy will be required to maintain system stability.

Users can use the 'quota' command to check their usage on gs18.

After the maintenance outage on August 4th, we are going to move a significant fraction of users from gs18 to a scratch space on ufs18. Affected users will be notified.

Webrdp is currently offline.  We are looking into this and will provide updates when available.


Update 10::00 am   The webrdp server is now back online.

We are currently experiencing a network issue causing most of our nodes to be offline.  We are investigating and will provide updates as soon as possible.


Update: 9:15 am  ITS has resolved a network issue in MSU data center all nodes are now back online


Update 11:15 AM: A core data center switch failed at 1:04 AM this morning due to a bug that switch's controller software. As part of a redundant switch pair a single failure should not have taken the network offline, but the second switch did not did not successfully take over. We have identified why the second switch was unable to bring up the interface and are working to implement a fix that will prevent this from happening again.

On Saturday, June 20th, at 8:00PM, all SLURM clients on the HPCC will be updated. This includes components installed on the development and compute nodes. As a consequence of this update, any pending srun or salloc commands run interactively on the development nodes will be interrupted. Jobs queued with sbatch, and srun processes within those jobs will not be disrupted.  Please contact us at https://contact.icer.msu.edu/ if you have any questions.

A recent update of the SLURM scheduler introduced a potential bug when specifying job constraints. Specifying a certain constraints may yield a "Requested node configuration not available error". If you encounter this error when submitting jobs with constraints that worked prior to Wednesday, June 10th, update your constraint to specify the 'NOAUTO' flag, e.g. 'NOAUTO:intel16' instead of 'intel16'. This will circumvent the issue while we work with our software vendor for a permanent fix. Please contact us with any questions at https://contact.icer.msu.edu/.

Update 8:47 PM: The scheduler is back online

We are currently working with our software vendor to address an issue with our job scheduler. The scheduler is currently paused. SLURM client commands will not be available and new jobs will not start until this issue is resolved.

      The slowness issue is resolved.

     One of the OSS servers on ls15 (/mnt/ls15) scratch file system is slow to respond to lock requests from the MDS server. We are working on replacing that drive at the moment. It will take a while to be complete. We will update this announcement once it is back to normal.

Today we will begin correcting file and directory ownership on all research spaces.  Please note this process will take up to several weeks to complete.  We will be contacting any users with large amounts of files which may cause research directories to become over quota before correcting ownership.

LS15 Maintenance

4/23/20 at 8am we will be performing maintenance to correct issues that are causing slow performance on our ls15 system.  The system will be slow to unresponsive during this time.  Maintenance is expected to be completed in less than two hours.

       Update on 03:05pm, the issue is resolved.

       ls15 scratch system (/mnt/ls15) is currently having a issue and we are working on it. We will update this information when it is back normal.

We will be performing emergency maintenance at 7am on Friday 4-3-20 to all gateways and development nodes.  This will require home directories to be taken offline on those nodes and the nodes rebooted.  We expect maintenance to be complete by 8 am

Update 4-2-20:  After patching the system a quota check has successfully run.  We believe currently that quotas reported are now correct.  

Important Note:  We are seeing about 50 research spaces over quota.  This is likely due to previous under reported quotas.  We have checked these with the DU functionality and they appear to be reporting properly.  Please remember that if you are storming large amounts of small files that the reported quota will not match DU due to system block size limitations.

Note:  We have ensured all default quotas are now enforced on research groups.  If you are having trouble with your research group please open a ticket and we will assist you.


      Currently our home file system check quota function will sometimes cause a users directory to have an incorrect quota. If you see this please open a ticket and we will work with you to temporarily increase your quota. We continue to work with our vendor to correct this issue.


Update 4-1-20:  We have received a patch and are testing to see if all quota issues have been resolved.   

Starting this morning we will be performing a patch upgrade to our home directory system.  This has been provided by our vendor to correct issues with quota functionality.  You may see some pauses in the system while components are restarted.  


Update 4-1-20 All maintenance is complete.

As part of MSU’s response to COVID-19, ICER is transitioning to an online-only support model. All HPCC services will continue as normal.


We are currently experiencing issues with our network that is causing slow or broken connections to several of our login and transfer systems.  We are looking into this issue and will provide updates as available.


Update 2/12/20 1:20pm  The issue is currently resolved.  We will be monitoring our network for further issues.


UPDATE (9:52 PM): The maintenance is complete and filesystems are remounted on the gateways

UPDATE: This outage is now scheduled for February 8th

On Saturday, February 8th, there will be a partial outage of HPCC storage starting at 8:00PM. This outage will begin with rolling reboots of all gateways and will interrupt storage access on the login gateways and rsync gateway only. This may cause 'cannot chrdir to directory' errors when connecting to the HPCC. Users can continue to connect from gateways to development nodes to conduct their work. Development nodes and running jobs will not be affected. This outage is expected to last several hours.

Hpcc login issues 12-24-19

At 10am today our gateways are no long able to communicate with our home storage system.  We are looking into the issue and will rectify it as soon as possible.  Compute nodes can mount the home system properly and jobs will continue to run properly.



From the week of December 9th through early January, we will be performing software upgrades on our gs18 and ufs18 storage systems, which will improve performance and reliability. During this time, users may experience periodic pauses and degraded performance. We will update this blog post if there are any specific impacts users may see and as the work proceeds. Please contact us if you have any concerns.

Update: 3:25 PM 12/20: After the upgrade on 12/19, two new bugs were introduced. Users may experience "Stale File Handle" messages, slow home directory or research space access, or not be able to log into a gateway or dev node when this problem is occuring. The vendor is preparing fixes for us to deploy today or tomorrow and we have an understanding and workaround of what's triggering this problem to reduce the impact on our users. We're sorry for any impact that this has on your research.

On Thursday, December 19th, the HPCC will be undergo scheduled maintenance. We will be applying GPFS software updates, adding redundancy for the Infiniband fabric, and additional minor fixes and updates. We will be rebooting every compute node, so any jobs that would overlap will be held until after the outage. The entire system may be unavailable during this work. We will update this blog post as more information becomes available.

Reminder to users: please be vigilant about data kept on scratch. Temporary data on scratch should be kept clean and updated to help prevent the file system from becoming full. 

Please see this blog post about additional file system work happening this December.

UPDATE: 6:20 PM The system maintenance has been completed.

Starting at 8am on 12-12-19 our vendor is performing updates to the software on our home system.  We do not anticipate any downtime associated with the upgrades.


Update 12-13-19 1pm: Updates have been completed on 4 of 6 storage nodes.  We anticipate the remaining storage nodes to be complete by end of day.  Updates on protocal nodes will continue on Monday 12-16.  When all existing equipment is upgraded we will be adding an additional storage block.

Update 12-13-19 4pm: On monday 12-16-19 users will see periodic samba home directory mount outages.

Update 12-17-19 8am:  Upgrade work on our existing home storage system is complete.  We will be adding additional storage to the system on 12-17 and 12-18.  During out 12-19 outage all compute node clients will have software updates to match the storage cluster.  During the outage we will also be replacing our AFM backup nodes with new hardware for better backup and over all system stability.

The HPCC is undergoing an upgrade of the GS18 scratch. No service interruptions are expected.


2019-21-12 all upgrades on scratch cluster are now complete

Today at 3:00 dev-intel16-K80 will go down for maintenance.  Available GPUs are not correct

and a replacement card will resolve this issue.  We will have the system returned to service as

soon as possible.

UPDATE:  Dev-intel16-K80 is working and available now.

We have a new AMD-based Rome server with 128 cores and 512 GB of memory available to users. It is currently accessible as eval-epyc19 via ssh from any development node. We are considering this architecture for our 2020 cluster purchase and would like your feedback on any strengths or weaknesses you notice.

We've configured 4 NUMA clusters per socket (16 cores.) In early testing, a hybrid OpenMP-MPI model that uses OpenMP processes per-NUMA-domain or per-L3 cache and MPI between processes provides excellent performance. You can see the system layout with

lstopo-no-graphics

This node is a shared resource so please be considerate of other users and aware of the impact other users may have on your benchmarking. If you would like exclusive access, please contact us so we can coordinate that.

Update (11:15 PM): The home file system is back online.


The HPCC is currently experiencing an unexpected outage of the home file system. We are currently working to resolve the issue.

After investigation we have found that quota enforcement for disk space usage has not been enforced properly.  We will be correcting this on 11-21-19.  We encourage users to check the disk usage versus quota and ensure that your research space is not over quota.  Looking at the current space usage about 30% of research spaces will be over quota.  We will be contacting the PI of each of the over quota spaces directly as well.  

Login Issue on HPCC Nodes

      09:55 AM:  Right now, there is a problem to log into HPCC. Please wait for more update.

      10:00 AM:  During the weekend, a home mounting problem happened for many compute nodes. The issue is fixed. 

      10:25 AM:  The login to HPCC is back to normal now. However, there is still a problem to log into dev-intel16-k80 node.

      10:55 AM:  dev-intel16-k80 can be log in now. The issue is resolved.

On Wednesday, October 23rd, the HPCC will be updating its installation of the Singularity container software to the latest version 3.4. This update adds new features including increased support for file system overlays and increased stability for containers using MPI. If you have any questions, please contact the HPCC at contact.icer.msu.edu.

10/17/2019 11:33 AM:   Most of the compute nodes are working. HPCC system is back to normal.

10/17/2019 10:03 AM:   There was a filesystem issue that has been resolved.  The gateway and development nodes have resumed full functionality. However, compute nodes are still not recovered.

10/17/2019 09:40 AM:   The HPCC is currently experiencing system issues.  We are working on problem and will update this message when we have more information. We are sorry about the inconvenience.


HPCC Staff



On Tuesday Oct 15 we will be adding new backup hardware to our home storage cluster that will be replacing legacy hardware.  As we add the new hardware the home directory system may be slow or pause at times as the file set backups recover.

How buy-in accounts are configured in the scheduler is changing. Buy-in accounts are currently configured with one partition per cluster, e.g. buy-in account “FOOBAR” with nodes in both the intel16 and intel18 clusters would have a “FOOBAR-16” and a “FOOBAR-18” partition. Buy-in accounts will soon have only one partition that contains all their buy-in nodes. This change will increase overall scheduler performance and will not affect how buy-in jobs are prioritized.

Rolling Reboots of Nodes

The HPCC is currently conducting rolling reboots of nodes to apply a file system update. This update will improve the overall stability of the GPFS file system. The HPCC will coordinate with buy-in node owners when rebooting buy-in hardware. These reboots will not affect running jobs, however, the overall amount of resources available to jobs will be reduced until reboots are complete.

Update (8:48PM): The SLURM scheduler is back online and accepting jobs

The SLURM scheduler server will be offline intermittently for planned maintenance on Thursday, September 19th, from 8:00 PM to 9:00 PM. During this time, SLURM client commands (squeue/sbatch/salloc) will be unavailable and queued jobs will not start. Running jobs will not be affected by this outage.

Home directory quota issues

We are aware that some user's quotas do not match what the du command displays. We have worked extensively with the vendor on this issue.  There are two root causes.

1) The quota check process would not complete properly.  On 8/20 we were able to perform a complete quota check which has corrected many user quotas.  We are still working with the vendor to insure this check can run successfully on a regular basis.

2) The new file system has a smallest file block size of 64k.  This means that files between 2K and 64K will occupy 64K of space.  This will cause disk usage for users with large amounts of small files to be inflated greatly.  We are working on a solution for this issue. 

     One suggested solution would be to add any files to a tar file, which would reduce the number of small files into a larger file. 

     A temporarily larger quota can be requested by a user if their quota is at 1T and they have many small files .

If you have questions or need assistance, please let us know.


At 8am EDT 9-4-19 we will be performing a ram upgrade on two of the controllers for our new home directory storage system.  We will need to move the system controller between nodes which may cause several minutes of degraded performance.  We do not expect any significant down time associated with this upgrade.  


Unexpected SLURM Outage

Update (12:15PM): The SLURM server is back online.

The SLURM server is currently offline. Client commands are not available, e.g. srun/salloc/squeue. New jobs cannot be submitted. We are working with our software vendor to find a solution.

We are currently having an issue with our virtual machine stack causing logins to fail and other systems to not work properly.  Our scheduler is currently paused and will resume as soon as the issue is corrected.

Update: 9:20 AM The issue has been resolved. Please let us know if you see any other issues.

The HPCC and all systems (including storage) will be unavailable on Tuesday, August 13th for a full system maintenance. We will be performing system software updates, client storage software update, network configuration changes, a scheduler software update, and routine maintenance. We anticipate that this will be a full-day outage. We will be updating this blog post as the date approaches and during the outage with more information. No jobs will start that would overlap this maintenance window. Please contact us with any questions.

Update: 10:30 AM The maintenance is mostly complete. We will be restoring access to the development and gateway systems shortly. We expect a return to service by noon.

Update: 11:20 AM. The scheduler was resumed at 11:15 AM and all services should be returned to production. Please contact us with any questions.



  • No labels