Gears server outage after failed reboot (RESOLVED)
-
Wednesday, 9th September, 2009
-
00:00am
We had upgraded multiple pieces of software and the kernel on the Gears server. It was supposed to be a simple reboot, which takes about 5-10 minutes. Unfortunately one of our technicians made a mistake and started the backup node which accesses the same file system during that time. This meant that two servers were accessing the same storage. This unfortunately causes file system corruption and were working to get those errors resolved. The system is currently running fsck and we will update this bulletin as soon as the issue is resolved. We sincerely apologize for the downtime and will issue SLA credits on request.
Update: 3:50AMCST
Fsck detected about 3% of the file system had issues, it was able to recover all of it and system is back up and stable. Thanks again for your patience.
