W3C - A major Breakdown

We experienced the most severe breakdown of Simcountry ever and ithappened in the past day.

Our main server, lost the main disks and took down the portal and Golden rainbow.

It took a major effort to fix the server and start restoring all functions.
The process took many hours of work and with the portal down, none of the worlds could be reached.

We have now restored the main functions of the portal and with it, 5 worlds are running.

These five worlds, (all except for Golden Rainbow) were running all the time but login is using the portal and the portal was down.

Now 5 worlds are running.

There are functions that are not fully restored.

Golden rainbow is not restored yet as we gave the portal the main priority.
we will now also fix Golden Rainbow.

We have never experienced such a breakdown, mainly because all storage is using two devices in paralel and they never break down at the same time.

now, it was the controler that failed and both storage devices failed.

We are very sorry for what happened.
we will look again into our procedures and backup hardware.

We will continue to repair and will restore all functionality ASAP.

We do not think that there was any damage to data but a small number of records, between the time of the last backup (once per hour) and the breakdown, could have been lost. This is only on the portal.

We have all logs and all transactions are secured.

Please let us know if you have any problem caused by this breakdown.

it could have been worse but it sill a relieve

Dear Simcountry Team,

I understand that you have experienced a severe breakdown in Simcountry, and I sympathize with the challenges you faced during the restoration process. To prevent similar incidents in the future and mitigate potential damage, I would like to offer the following advice:

1. Implement Redundancy: While it is commendable that you had two storage devices in parallel, it is essential to ensure redundancy at every level of your infrastructure. This includes duplicate servers, network connections, and power supplies. Redundancy helps distribute the workload and prevents a single point of failure from bringing down the entire system.

2. Regular Testing and Monitoring: It is crucial to continuously test and monitor your systems to identify any potential weaknesses or vulnerabilities. Regular load testing, stress testing, and security audits can help identify areas that require improvement. Additionally, implementing proactive monitoring tools can provide early warning signs of potential failures and allow for timely action.

3. Disaster Recovery Plan: Having a robust and well-documented disaster recovery plan is essential in mitigating the impact of system failures. This plan should include regularly scheduled backups, off-site storage, and clear procedures for restoring services. Testing and simulating disaster scenarios periodically can help ensure the plan's effectiveness and identify any gaps in the recovery process.

4. Regular Hardware Upgrades: Aging hardware can become more prone to failures. It is important to regularly assess the performance and reliability of your hardware components and consider upgrading or replacing them when necessary. Staying up to date with the latest technology can help improve the overall stability of your systems.

5. Communication and Support: During incidents like these, clear and timely communication with your users is crucial. Providing regular updates and reassurance on the progress and expected recovery time can help manage user expectations and minimize frustration. Additionally, having a dedicated support team available to assist users and address any issues arising from the breakdown is essential.

I hope you find these suggestions helpful in preventing similar breakdowns in the future. I understand that damage has been done, but by implementing these measures and continuously improving your infrastructure, you can minimize the risk of such incidents occurring again. (The GR world is still down at the moment) can't access my main country. The booster functions/tabs are down also.

Please let me know if you need any further assistance or have any additional questions.

Best regards,
Greg

Golden Rainbow is now also running.

There remain some problems.
the forum is not fully functional.

we will now continue testing and resolve remaining issues.

Greg how long did it takes chatgpt to spit that out?

Golden Rainbow Have not move one day for about 24 hours now.
It's stuck on 27th of september 5058

@andi

The forums are not loading on pc
only on mobile
Please look into this

Forums back on PC.

While Prince Gregs comments seems excessive as we are not informed of how the back-end of the game is managed and you likely do most if not all of those things, I do agree with Prince Greg on point 5, the dev team should have had someone posting a simple comment on what was happening on the game discord, and ideally had a backup landing page to redirect the domain to with a simple message of the issue.

Note: This doesn't need to be constant messaging and answering questions, just a quick message stating you know there is a problem and are working on it, with perhaps updates on estimated time until recovery.

Andy,

Something you might want to check out regarding product storage.
I am testing it now on "Melchior" on LU.

I've been buying wind turbines for one of my nations it seems like there is a limit of 996,000 turbines that can be stored within a country. The country will continue to buy over that limit, but any purchased seem to "disappear".

I'm testing it to make sure it is not some kind of weird coincidence, but as of now:
The stock in the country is 996,001 turbines.
I use 1000 per month
I have ordered 10,000 more.

Within the next 4-8 hours I will follow here. If it is miscounting then there is no need to check, if there is an issue I will let you know. That way you can fix anything quickly.

Andy

It is not processing my game months. No worries but wanted to share

UNSF on Golden Rainbow btw

I think we should be compensated with Gold coins. I was at war when this happened.

Andy,

So I did the math and something doesn't add up

---------------------------------

As of "Jun 21, 5072" on LU my nation Melchior has exactly 1,000,000 wind turbines with 516 waiting to be delivered.

On "Feb 5, 5072", I had 995,001 in stock with 16,466 ordered.

Based on the fact that I went from Feb to June, I have should used 5000 turbines in consumption.
I had supposedly 16,000 delivered within the last few months. It seems that there is a limit to how many wind turbines a nation can have. It seems that is stops at 1m then the rest of the turbines disappear.

I'm not mad or asking for a refund, but I have been mass ordering for awhile thinking I was stockpiling when it seems they keep disappearing.

Might want to check to make sure no other such limits exist on other products

Andy,

I give it one game month. I also have screenshots.

So my country bought almost 2800 wind turbines on July 6th 5072.
Last month I had 1,000,000 turbines

I should have 1,001,800 turbines (minus 1000 for consumption). Yet my country stock is 999,000

Like I said might want to check out the limits so this does not happen to other people or products.

Andy, great job getting our countries back up. But my profits fell dramatically after the date it froze and have stayed low, anything you can do to help there is appreciated.

thank you for getting the game going again.

space transports aren't really working

it will freeze while trying to set up the transport. usually i can get one transport going, then have to reload the entire thing, and do one more. sometimes i can get 2-3 before it freezes, sometimes i have to try 5 times to do one

i can't maintain my TA empire like this. i appriate your efforts to keep the game going. but i'm gonna have to pull from TA if this doesn't change

We had a complex problem with multiple disk failure.
we never had such a breakdown that continued despite extensive backup procedures and multiple copies of everything.

we did not have any damage except for delays in processing.

We have put Golden Rainbow on 8 months per day to catch up.
It is now back to normal.

we see more smaller problems that will be resolved in the coming days but processing is running correctly.

One of the problems is that the server running the space transports is heavily loaded.

the problem is resolved but there is an issue that will be resolved on Monday.

In the mean time we are keeping an eye on it and make sure space transports are on schedule.

I am having the same issue with shuttles.

One or two shipping requests lock out the screen,

Have to restart to finish.

Thanks Andy,

Thanks for all your hard work Andy! You and the team are amazing.

Same as Johanas for me. It's sort of locking simcountry entirely when I attempt transport requests.

I have looked into the transports.
the server might have been overloaded during a short period.

I see all transports running OK.

Please mail the gamemaster if you see any problem.

https://www.youtube.com/watch?v=h5lJTcChkoA

worth a 3 minute break

and thanks for keeping it going