Getting ready for highly available SQL Server system, is something that client wanted to achieve for the server that they have from recent acquisition. Before we were able to complete the review and prepare what we need to achieve we had a page that their server was crash and it was an emergency. The server which was crashed was sitting on top of Always-On and was running on virtualization as part of the consolidation done some years back. The server that we had been paged for was the one we were reviewing from configuration point-of-view, and we have copied the database backup files to another server (the UAT box) for testing.
So I and my colleague were kind of thinking that we will be able to bring up the application and production as soon as we do the health check of secondary server. While on the call we have noticed that the secondary server is not available as well, and, the backup disk was gone too. That is catastrophic failure!!. We were able to survive with a day old backup data that we have saved on uat box for testing purpose. Let me share my observation with you, which can be treated as tips as well. If you really want to be getting ready for highly available SQL Server system, please remember to follow the tips I have mentioned below and while you read this, I would encourage you to please review my previous post about disk :
Findings
- The SAN was crashed actually
- Both the SQL Server is hosted on a single VM. this has caused the main issue. When it goes for toss, both Primary and Secondary server was gone.
- Backup disk was gone too, because data, logs and backups disks were on the same disk
Recommendation
- Probably you may ask your SAN admin to replicate SAN storage
- Never host Primary and Secondary server on same VM and same Storage
- Have your backup storage on separate SAN disk
photo credit: Claude-Yolande Décrocher la lune via photopin (license)