[Resolved] Server Maintenance - 3/12

Update (11:00 PM CET - 18/12):
s2.filefactory.com is now back online and all data was successfully restored / transferred to other servers.  We apologize for the extended downtime caused by this issue, and will ensure that disk verification checks are performed more proactively in the future to minimize downtime caused by issues such as this one.

Update (09:40AM CET - 12/12):
There are further issues with this server, which are causing the array verification to fail.  We have decided to transfer the affected data off this server onto other download servers, so files that were hosted on it will progressively become available from this point forward.

Once the data has all been transferred we will re-initialize the server and replace the hard drives, to avoid this issue reoccurring.  

Update (09:50PM CET - 9/12):
The verification process is ~65% complete on s2.filefactory.com.  We had to replace one of the RAID cards which is why this has taken so long to get resolved.

We are hoping that we will be able to bring at least 50% of the data on this server back online early tomorrow morning, and the remaining 50% before the end of the week.

Update (06:15AM CET - 8/12):
We have solved the issue that we encountered when working on s2.filefactory.com and are now in the process of verifying the disk arrays.  We apologies for the continued downtime of this server, but our priority is to ensure that data loss is avoided wherever possible, and this verification process helps with that.

Update (10:00AM CET - 4/12):

Unfortunately there are no further updates on s2.filefactory.com yet.  We are continuing to work on it, and as soon as an update is available this post will be updated.

Update (1:15AM CET - 4/12):

The following servers are now being brought back online:

  • s5.filefactory.com
  • s10.filefactory.com

An issue has been discovered with s2.filefactory.com, so that server will remain offline while it is investigated.

Update (12:55PM CET - 3/12):
The disks have been replaced and the verification process has been started on each server.

Original Post:
Starting at 8AM CET today (3/12) the following storage servers will be temporarily taken offline for urgent maintenance:

  • s2.filefactory.com
  • s5.filefactory.com
  • s10.filefactory.com

We will be replacing several failed or failing hard drives in each server.  We have opted to perform all of the replacements at once to minimize downtime and attempt to reduce the inconvenience caused by this critical maintenance.

We will endeavor to get the servers back up and running as soon as practically possible - please note that we will need to verify the integrity of each disk after it's replaced, to ensure there is no data corruption.

This article will be updated when more information becomes available.

Have more questions? Submit a request

0 Comments

Article is closed for comments.