The database technology of Guild Wars
I have the enviable task of researching online game and virtual world technology. My first interview, quite naturally, was with the lead developers of a game I actually play – Guild Wars. The overview is in another post; that may provide context for this one, which focuses on the database technology. (I also did a short post just on the implications for Guild Wars players.) It also has a brief description of what Guild Wars is – namely, a MMORPG (Massively MultiPlayer Role-Playing Game) with the unusual feature that most of the game world is instanced rather than utterly shared.
First, some scope. ArenaNet (Guild Wars’ developer, now a subsidiary of NCsoft) runs Microsoft SQL Server, mainly Enterprise Edition, having just switched to 2005 4 months ago. They run 1500-2500 transactions/second all day, spiking up to 5000 in their busiest periods. They have no full-time DBA, and when the developers started this project they didn’t know SQL. They’ve only had one major SQL Server failure in the 2+ years the game has been running, and that was (like most of their bugs) a network driver problem more than an issue with the core system.
As for what’s going on — there are a few different kinds of database things that happen in an instanced MMORPG.
- Game characters acquire loot; i.e., there are inventory transactions. These are commonly one-sided. Goods are created pretty much out of thin air – or rather thin electricity – and so there’s no notion of matching a deletion from stock against consumption in manufacture or anything like that. Similarly, they buy and sell at merchants and traders and so on. One character carries around 60 different items, each of which is a combination of several tokens or attributes chosen from a finite list.
- Characters evolve. They gain power. The portion of the game map they can see increases. They experience milestone events in the game. Etc.
- There are temporary changes to the game world as characters explore areas. These need to be maintained as state while the character is online and in the specific area; but because the game is instanced, they don’t need to be persisted beyond that (with minor exceptions).
- There are a variety of secondary systems with ordinary transactions, such as guild rosters and tournament standings. Based on their functionality, it would seem these have extremely simple schemas.
In some MMORPGs, inventory is handled in the obvious way – each item has an associated record. But Guild Wars does things differently. Everything about a character is one big BLOB (Binary Large OBject), typically 10K-30K in size. (One component is the whole game map, or at least a record of which pixels the character has opened up.) More precisely, there’s a character BLOB and an account BLOB. BLOBs are updated as the game progresses, then saved every few minutes.
Actually, if there are are 150,000+ transactions per minute, and each character generates a transaction every few minutes, that suggests there are ½ million or so characters online at once, peaking at 1 million. Based on what I know about gameplay and so on, I think that figure is too high by a factor of 10 or so. I can’t immediately explain the discrepancy.
Of course, there’s almost no simpler kind of transaction than banging a BLOB into a database. Even so, that’s a lot of BLOB-banging. So between the game server and the database backends, there are custom cache servers. The game servers send messages about various kinds of structured game-object transactions to the cache servers; the cache servers then bang updated BLOBs into the database. These cache servers are extremely stable, staying up 150-200+ days at a time.
ArenaNet is, on the whole, quite responsive to player requests for upgrades. But in one regard they’ve consistently disappointed people – they have omitted introducing an ingame auction house, something competitive MMORPG makers seem to be able to build. Even after the interview, I’m not 100% clear on the reasons, but I think they mainly boil down to this – an auction house would be by far the most complex transactional system they’ve implemented. And given that the inventory isn’t currently handled on a record-by-record basis, it pretty much would have to be built from scratch. That said, while they gave me no hint of this — when they reexamine the issue for Guild Wars 2, I bet they’ll realize it’s not as hard as they now think.
As for other database futures – just taking account (as oppose to character) inventory from 20 to 80 slots required database scale-out. More database scale-out in the future is likely, although they haven’t firmly decided on that yet. ArenaNet seems very happy with SQL Server, and seems unlikely to fix what isn’t broken; besides, they’re based in Bellevue. (By way of contrast, Sony Online is in the process of moving Everquest from Oracle to Enterprise DB. I hope to interview them in the future.) ArenaNet doesn’t seem to be considering memory-centric technology, although that seems an obvious choice for caching; in particular, StreamBase seems to be getting a little traction in the game/virtual world market. (More on that later if the StreamBase folks help me get some interviews.)
Related link
- A mid-2009 post on the database technology of Lord of the Rings Online and other MMOs.
Comments
14 Responses to “The database technology of Guild Wars”
Leave a Reply
Great articles very well thought out and written… Thanks for taking the time and looking forward to more of your work in the futue
Very interesting, good read. I guess I never thought about it, but I never thought about commercial games companies using SQL based databases, thought they’d use an in-house thing. Makes sense though if I think about it.
[…] technical facts about Guild Wars from these three […]
Thanks for the read. I’ve been playing GW for a couple of years and, as a DBA, have wondered about the technology behind the scenes. I’m surprised about the BLOBs, given some of the limitations in character gear, but hey, whatever works, right? 150k transactions a minute…. that could be right. Assuming 10% are online at any point in time, and half of those are out killing critters, that’s only 1 person killing 1 critter in 1 minute. It’s hard to run for 1 minute without running into a group of critters, so that seems plausible. It could be higher, actually, given that there are people that “solo” or “farm” areas, trying to kill everything by themselves as quickly as possible.
Bour Gie (L20 Ranger/Ritualist)
No. There’s definitely a problem with my numbers, or my understanding. The key point here is that there isn’t automatically a transaction every time a critter is killed.
And of the 1-2 million GW players total, not all that many of them are online and actively killing things AT ONCE.
I’m kind of curious as to if instancing was also one of their solutions to deal with possible
transaction collisions. Curt do you know if collisions are minimized due to instancing or not?
I’m sorry, Stephanus, but I don’t understand the question. I’ve never played a persistant/non-instanced MMORPG, but don’t see where collisions would come in in either case.
In a persistant/non-instanced MMO there can be a heavy load on servers, which would possibly cause a
slowdown in the rate of data transfer(killing a monster, grabbing loot, buying/selling/trading, etc.)
or even data collision inwhich the transaction is not completed. I have experienced this in Final
Fantasy XI, but not so much in Guild Wars. So, I was wondering if instancing played a roll in
minimizing these types of occurrences.
How many players share a server/slice in a persistent MMO? Could Guild Wars’ BLOB strategy even work?
The developers did say that the BLOB strategy helped with transaction integrity, against some exploits to deliberately crash the system mid-trade and hence duplicate items.
That said — it has to be possible to maintain transaction integrity in a persistent world too. We’re not talking THAT many individual transactions, although again it depends on the number of players per server.
[…] views (DMV) of SQL Server 2005. Curt Monash wrote about how you can use MSSQL Server 2005 behind a Massively MultiPlayer Role-Playing Game […]
[…] a research project on the IT-like technology of games and virtual worlds, especially MMORPGs. My three recent posts on Guild Wars attracted considerable attention in GW’s community, and elicited […]
[…] last point is not a joke. One of the weirder database architectures I know of is the one underlying Guild Wars. Its developer — a brilliantly impressive guy — told me flat-out that he learned in […]
[…] blobs can start out as being unassuming and small, but can quickly become a significant problem. Kurt Monash at dbms2 provided a good example of the use of Blobs for object storage in the context of the Massively […]
[…] in the world, they’re at least on the short list for consideration. The makers of Guild Wars didn’t even try to have decent database functionality. A decade later, when they introduced Guild Wars 2, the database-oriented functionality (auction […]