RIT SE Systems (@RITSESystems) profile picture

RIT Software Engineering Systems

Joined May 2013
SE email is flowing once again. The mail server has been moved to new hardware.
All SE systems should be back up after the electrical outage.
1
Apparently one of our UPSs had a power issue. That UPS contained one of our VM hosts so nitron, control, the website, and some other support machines were down. I'm checking on their recovery now.
All OpenBSD/UNIX servers are back up and going using the new file server.
OpenBSD (unix) servers will go down at 8:30pm for the cutover to the new file server.
Mightor went down with the power and three of our VM hosts did not deal well. Reboots are fixing the problem.
Power blinked in the building. One of our UPSs seems to be bad. Assessing the damage.
The network jack for one of our VM hosts went dead and disconnected all its VMs from the network. The host has been moved to another network jack and all services are back online.
The mail submission server was upgraded earlier and there were a couple bumps. Mail is flowing normally now.
nitron didn't want to communicate with mightor after mightor's crash, so I restarted nitron. Service is restored.
All services are restored and seem stable. Have a good night.
Board is swapped. Verifying service restoration.
Systems downtime starting in 5 minutes.
Systems will be down starting at 8pm tonight in order to install a new control board for the file server. I expect the whole process to take less than 2 hours. Updates will be posted here.
Well, after being up for more than 48 hours and giving hope the problem might be solved, mightor has crashed again. This time the vendor and I should have more information to find the cause.
Mightor has panicked three times so far today. I'm working on it.
1
Windows shares are temporarily disabled while the problem is worked on.
1
Windows shares have been re-enabled. System is staying up. I think we have a root cause. Just need to verify things.
There seems to have been a brief network interruption. It is being investigated.
It's unclear what happened but nitron went unresponsive. Ultimately I had to hard reset the VM. Things seem back to normal now.