At 7am 5/26 several nodes went down hanging our home directory system. This is a know bug in our GPFS system and we are waiting for a patch for our nodes. At 5:30pm we were able to locate and restart the culprit node, and resolve the issue.
05/25
Update 5:30 pm All research spaces have been migrated. Two active user are still with very large file counts are still syncing.
05/24
Update 5:30 pm We continue to migrate users, this has been slowed due to issues with the home file system earlier tody. Currently 5 active users remain to be migrated and a single research space.
05/23
Update 10:40pm The remaining 14 active users are still syncing,…
We are working on an issue that has caused all of our nodes to go into an offline state, as well as not allow users to login. We will have the nodes back online as soon as possible.
Update 10:26 Nodes are back online and running jobs. Logins at this time may still fail.
Update 12:00 We continue to experience issues with mounting home directories, which is causing gateways to be unresponsive. We are working with the vendors to resolve this issue.…