As you may or may not be aware, Hightrees Organisation Limited has recently suffered hardware problems with the server which we use to host websites and provide some clients with email facilities.
Below is a timeline of the issue from start to finish
Wednesday October 29th circa 23:54
The web server was discovered to have faults on reading/writing to the hard disk, so the system was taken off line.
Thursday October 30st 8:00am
Contact was made with the hosting company to establish why the none of the services hosted by Hightrees Organisation Limited were available. We were updated with the issues that had occurred overnight.
10:30am
A replacement server was on site and the core operating system was being installed.
2:00pm
Server was briefly online whilst the operating system was installation was verified and firewall services were
configured.
6:00pm
Underlying Web hosting Software, Database Applications and additional services (mail, FTP etc) were being
installed and configured.
10:00pm
We were informed that the server would be on-line in the early hours of Friday morning. Data recovery would be started as soon as possible once the server was fully online.
Friday 31st October
00:30am
We had limited access to the server, but verified that core services were up and running. Temporary workarounds for certain email systems whilst waiting for the system accounts to be restored.
07:00am
Some basic accounts and services hosted by Hightrees had been restored.
10:00am
The failed hard drive was being analysed by hard drive specialists to see how much data could be recovered. This is a very time consuming process as the drive is examined at a low level, using specialist
software.
Saturday 1st November
02:00am
Installed empty holding pages for those sites still offline whilst waiting for results of data recovery.
Monday 3rd November
10:00am
Hard Drive contents had been examined, showing 40,000 files are technically available, these need to be
sorted into which data files belong to which accounts.
21:00pm
Attempted recovery of an old backup failed, due to corruption in file. This is most likely due to the hard disk
starting to fail and corrupting minor data before failing completely.
Minor configuration changes made to server with regard to setup of some services.
Tuesday 4th November
Waiting for an update with regarding to alternate data recovery method.
Friday 7th November
Those files that could be restored have been, however due to the level of corruption this has been fairly
limited. Any data which has not already been restored is unlikely to be recovered.
Outcome and Resolution
Hightrees Organisation Limited is not happy with the recent problems we have experienced or the loss of our customers data, and whilst we have tried everything feasible possible to try and recover any lost data with our service providers this has not been possible in some cases.
As these problems have have been beyond our control, we are actively looking at the best solution to prevent these type of problems occurring again. As such we are looking at implementing a backup service, which is setup in such a way that we are in full control of the backups and their storage. These are likely to be held on an alternative secure server on a completely different internet service and are monitored on a daily basis with reports being emailed to a designated contact within your organisation/company.
We can only apologise for the inconvenience and we will be sending full details and summary of the backup services out as soon as we have finalised the details.
