Tue, Nov 22, 2011
Around midnight, fs4.cs, one of our NFS fileservers, failed for reasons that we are still investigating. This stalled all access to its filesystems, which had various bad effects on other machines. Among other things, our mail machine stopped processing email, the IMAP server became very unhappy with life, apps3.cs crashed at some point, and apps0.cs (aka cs.toronto.edu) stopped responding to ssh logins due to being overloaded.
When core staff arrived here this morning, they were able to revive the fileserver (in the end it required deploying new hardware), return fs4's filesystems to life, and get all of the affected machines back in service. We believe that everything should now be back to normal.
There may be minor disruptions to fs4 later today, because we believe that one of the disks it uses is failing and needs to be replaced.