W3C - Server failure Wednesday early morning.

Very early on Wednesday, November 5, at about 1 am, our main server failed.
it was a major breakdown and it took us several HOURS to fix.

we are sorry about it.

It was fixed without any damage to data.

Unfortunately, we are in the dark on the reason for the failure, we are running all possible test to try and find out.

We hope that it was a one time problem.

The server that failed, runs the portal, Golden Rainbow and Tiny Atlas and the failure stopped the entire site.

Heads up, GR is still stuck on Dec 31st 5419. LU&WG are still moving.

Heads up, GR is still frozen in time.

He's right either that was the fastest year in Simcountry history or Dec 31's is the longest day.

bei mir steht immer noch 31 dezember

We had a major failure on the main server and were able to fix it with no loss of any data.
However, it took quite some time to fix, and repeated on/off availability until all was done.

we are very sorry for this problem.

we keep extending our backup and reserve capabilities to prevent such cases.

The site returned to full functionality on Thursday around noon EST.

we are currently running database checks on Fearless Blue. everything is OK but it can take several minutes before this procedure is completed.

Must have been a long couple of days! Thanks again Andy!

GR is still having simulation problems. It was moved up to May 12th 5420 when when I logged in, but nothing changed. My economy hasn't made any transactions and all values are still the same as yesterday.

GR is now stuck at May 31st 5420. All game parameters and values have remained unchanged since Dec 31st. 5219.

This failure was quite complex and caused several major system stops because it happened on the server that is running the portal (and Golden rainbow).

luckily, such failures do not happen frequently.

it required us to go to the place where our servers are housed.

It was out first visit there in more than a year.

I thought you just bought new servers and had to move to a new building?

What I got from what Andy said here: They upgraded to new servers because their site host was closing or moving or something. Then they moved the new server to another site.

It seemed a little backward to me, as I would have sent the new server to the new site and then just move the files to the new site, after everything was working.

But I guess they had more than one server, EG they had actually only replaced one of several.

It was fuzzy.

The brief return of nuclear disasters was exciting though.

The site we use is closing on December 31.

We have contracted a new site and are going through the procedures to get the IP numbers we need and all resources needed to run our servers.
This phase is now completed.

we get the new site on December 1 and will bring there one spare server to test.
if all works, we will move our operational servers to the new site in one of the following days, early morning.

To prepare for it, we have indeed installed some new servers on the old site and gradually moved the worlds to the new servers.
We did now replace everything. Some are quite new.

Then, after all was done, we had a failure that caused the disruption.
This was unrelated to the moves that ended some time before.

we now know exactly what happened and made changes to try and avoid any repetition.

The new servers were purchased some time ago and were installed on the "old" site but did not participate.

Then we installed all the software and started to use them.

in the process we retired 3 servers, the oldest ones.

I hope it is clear.
we will of course try to make this move to the new site as fast as possible.

During the initial transition period, we also considered a possible move to the cloud.
However, Simcountry is using a lot of computing power to process a huge number of daily transactions.
The computations are processor bound and need dedicated servers that will be available to us 100% of the time.
This is not a typical application for the cloud.

Andy, please keep us updated on all the technical details you care to share. Some of us find it fascinating to see behind the curtain.

If you don't mind sharing, what were the specs of the old servers? I'm guessing some pretty impressive Sandy Bridge era Xeons?

We are using Dell servers with many XEON processors and 72 to 100+ cores.
The servers have 128Gb to 192Gb of memory.

All disks are SSDs. ranging from 512Gb to 4Tb. very fast.

All SSDs are using raid 1 which means that they are mirrored.
Everything is stored on two identical disks and in case one fails, we can replace a disk without even stopping the server.

Processing is in fact processor bound.

We run several virtual engines on each server.
These have 16 to 24 cores, 32Gb of memory and several hundred Gb of disk space.

The memory size is large enough to keep the entire database in core.

We use Fedora core operating system. (It is all Unix based).

Month runs including trading (6 times per day) conclude within 50 minutes to 75 minutes, Some worlds are larger than others.

The portal can easily handle the traffic, and the space transports even when there are 1000+ shuttles moving.

Two reserve servers are also installed.
These are turned off but can be turned on remotely and replace one of the other servers if needed.

Recently, we used one of these servers.