| Andy | Friday, September 20, 2024 - 12:40 pm 
We had some problems with trading on White Giant.
similar problems appeared on LU before that and on FB too.
The problems started when corporations started selling some of their stored products and at the same time, some players too were dumping large quantities of products.
It coincided with a large increase in visits and clicks on our servers.
The database tables that store trade transactions have a limitation in their capacity.
Mainly the part that holds a log of each and every transaction that takes place in each country.
Such trading surges can take place at any time and can add trillions of data in short periods.
We plan a solution for the problem but it will probably take some time to complete.
A short term solution could double the capacity and will probably be sufficient. we will implement it ASAP.
In the mean time, we have added some alerts that will warn us that such a situation might occur, and when we are alerted, we can easily resolve the problem.
WE are sorry about this.
Our servers are now processing much larger numbers of transactions than before as the numbers of logins and clicks is growing fast. |
| Laguna | Friday, September 20, 2024 - 02:51 pm
Maybe we can reduce the stored data as well.
For instance, looking at my country's purchase log:
| Entry | Product | When | Quantity | Quality | Price Paid | Price | What happened? | | 1 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 880,166 | 120 | 2,102 SC$ | 1,752 SC$ | Market Sale executed. | | 2 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 1,147,936 | 126 | 2,224 SC$ | 1,752 SC$ | Market Sale executed. | | 3 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 1,122,677 | 120 | 2,102 SC$ | 1,752 SC$ | Market Sale executed. | | 4 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 1,122,676 | 120 | 2,102 SC$ | 1,752 SC$ | Market Sale executed. | | 5 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 1,106,962 | 120 | 2,102 SC$ | 1,752 SC$ | Market Sale executed. | | 6 | Books And Newspapers on Kebir Blue Market | Jun 04 5450 | 1,045,020 | 129 | 2,263 SC$ | 1,752 SC$ | Market Sale executed. |
Entries 1, 3, 4 and 5 have the exact same characteristics (product, date, price, quality, and status). From my end, they could all be summed together and no information would be lost. But is the reason we have this level of granularity because we actually have a filtered view into the transaction log of everything that happens in game? So these entries are actually used to relate buyers and sellers, even if they are anonymous above.
By the way, the table above is copied almost exactly from the game interface, so the  suggest it uses a non-standard character. I'm not sure if this is in the db too, or just the GUI. Maybe some optimization can be done on that too. |
| Laguna | Friday, September 20, 2024 - 11:22 pm
I'm aware you have decades of experience on this, predating even the game, but my guts tells me there's something more here. I don't count on solving anything here, but help me if I am mistaken.
The transaction log above suggests we have a detailed trade history of A selling to B. The quantities above match with a single corporation's monthly production. The names are just anonymised behind Kebir Blue Market.
I like this approach because it is a point-to-point matching of demand and supply.
However, if we introduce a Central Counterparty, where ALL parties sell to the CCP, and ALL parties buy from the CCP, we are able to aggregate transactions with the same characteristics. From the player's point of view, the introduction of the CCP is invisible. In terms of storage it should prove to be less demanding.
In this scenario, the challenge of using a CCP is building a matching algorithm that works with this efficiently. I suspect it is possible, but I'm not seeing it. Plus, I already have a hard enough time dealing with a three dimensional problem (time, price, quantity), extending this to four dimensions (quality) is more than I'm willing to entertain. I suppose dropping quality from the equation, and just allow the CCP to adjust quality as needed to both the sellers' and buyers' end would be a way to reduce the problem to a standard one.
I know you just wanted to explain the situation, Andy, but I just needed to air my thoughts.  |
| Andy | Thursday, September 26, 2024 - 04:35 pm
Laguna,
Most times, I mean 99.999 of cases, the log as created by the trading is large but not huge. about 10% of the current max.
we are logging each and every transaction.
If a country orders 10.000.000 units, there are probably many sellers as corporations produce smaller numbers and the order is being delivered by many sellers.
The offers and orders are stored in the database as they occur and are then processed according to the trading strategy and priorities, qualities and prices.
It works fine. we have no problems there.
The transaction log is stored in daily tables.
This is done to prevent cleanup operations. we just remove the oldest table each day.
There is a technical issue with the size of these tables.
We bumped into it because of the resolution of an error that stacked up a lot of products in corporations.
To make sure it never happens, we will either resolve the technical issue which requires a serios change to a basic library, or just create 2/3/4 tables each day which is simple and will make the problem go away.
The classic way this is done in the "industry" is by a cleanup function
that removes older records.
This is, in this specific case, a massive action that will take for ever.
Removing records is relatively a time consuming function.
removing 1.000.000 records is a different story.
We have more than 20.000 countries, hundreds of thousands of corporations and millions transactions each game month.
there are probably some banks with much smaller numbers. |
| Laguna | Thursday, September 26, 2024 - 06:39 pm
Always nice to have a peek inside the box.
I like the solution of deleting a whole table, instead of deleting old records.
Good to know the usual size of a table is about 10% of the max too. |
| Boutros Boutros Boutros-Gunnarson | Thursday, October 3, 2024 - 08:27 am
Trading is not functioning on WG as of right now. |
