The issue that caused the mid-December outage never had anything to do with hardware, even though the FA administrators attempted to move from one machine to another attempting to fix the issue.
This was, start to finish, a software database problem; specifically, the database portion that generates notifications per-user in the upper right corner (E.g. 1020S 105C 41W 2N). Those notifications never expire for any user, and the software backend they are stored in is inefficient, meaning that millions of submission/comment/etc. notifications that will never be read or used have been choking it.
The real solution to that problem is culling notifications, and putting the notifications framework into more efficient software. No amount of hardware will solve that problem. In fact, until yesterday (12/19), the 'Nuke all submissions' button had been disabled because of the high load it was causing the server.
(In fact, based on what I've read, the hard drive failures - particularly the ones on Monday - were probably caused BY the data transfers back and forth torturing the disks. One FAF admin site status post said as much.)
The issue that caused the mid-December outage never had anything to do with hardware, even though the FA administrators attempted to move from one machine to another attempting to fix the issue.
This was, start to finish, a software database problem; specifically, the database portion that generates notifications per-user in the upper right corner (E.g. 1020S 105C 41W 2N). Those notifications never expire for any user, and the software backend they are stored in is inefficient, meaning that millions of submission/comment/etc. notifications that will never be read or used have been choking it.
The real solution to that problem is culling notifications, and putting the notifications framework into more efficient software. No amount of hardware will solve that problem. In fact, until yesterday (12/19), the 'Nuke all submissions' button had been disabled because of the high load it was causing the server.
(In fact, based on what I've read, the hard drive failures - particularly the ones on Monday - were probably caused BY the data transfers back and forth torturing the disks. One FAF admin site status post said as much.)
Further evidence that these problems are about software and not hardware: a Friday 12/20 restart. http://forums.furaffinity.net/threads/867077-Server-restart-on-Fri-2013-12-20