Page 1 of 5 123 ... LastLast
Results 1 to 10 of 45
  1. #1
    Join Date
    Apr 2011
    Posts
    11

    Lee & Nelson: Some of my terrain persistence research...

    Lee,

    Having been CTO of a software company in a former life, one of the lessons I learned the hard way was, "If you code it, you have to maintain it, too."

    I like the idea of coming up with our own storage system, but I like the idea of finding one that already exists better.

    I haven't done any testing, yet, but here are some of the candidates I'm looking at:

    Membase. Apparently, this is what drives Farmville and the other Zynga games. I know they don't have anywhere near the storage requirements we do, but this thing looks like it will scale almost infinitely (by adding nodes). Also, it's not a sophisticated object database, it's just focused on storing key-value pairs. I can't help but wonder if that's not a perfect fit for a grid of patches.

    Redis. This is another key-value focused data store. It keeps all of the keys in memory, and swaps the values out to virtual memory. Initial research says this will scale big, too, but it's not as proven as membase. Interestingly, this guy was just hired by VMWare and they're paying him to continue working on it.

    Versant. This is an enterprise-level object database. We used this at my old workplace, to persist the data for our custom web development system. Again, I haven't done any testing with a scenario like ours, but I remember the Versant sales guy emphasizing to us how well they could scale. He was saying they handled things like stock exchanges, telephone call routing systems, etc., with ease... and those systems have IMMENSE storage needs. It's a free download during development, then $5,000+ (depending on how many nodes, etc.) for commercial use.

    Anyway, these may not help at all... or, maybe one of them will be the exact thing you're looking for.

    If you have any suggestions for how I can test these, what kinds of things a test script would need to do, I'll be glad to report my findings!

    Regards,
    Jeff

    In case anyone is following this thread, here is an interesting discussion I found on membase scalability:

    http://groups.google.com/group/memba...af69cb4bd5e6c2

    As I understand it, a zone in our game is going to be 801 x 801 patches.

    Since everything in membase is stored as key-value pairs, we could store each "blob" of data for a patch (around 326KB) with a simple key like, "1:542:761" where 1 = zone, 542 = patch X, and 761 = patch Z.

    Each key is about 9 bytes, working out to 5,774,409 bytes per zone, or 5-6MB. That's how much RAM we'd need to keep the keys for a single zone in RAM (plus whatever is required for membase's metadata.)

    The slick part about membase is that it's reliable, guarantees data integrity, is easy to backup, and is stupid simple to scale. You just fire up more nodes on EC2 or whatever other cloud service you're using, add them to the storage cloud, and you're done.

    Very interesting indeed.
    Last edited by fatgav; 05-02-2011 at 08:02 AM.

  2. #2
    Join Date
    Dec 2002
    Location
    Virginia Beach, VA
    Posts
    861
    The problem with these types of solutions is that while they may work for the server side, how does the client cache their local copy of the data? Unless you propose to have them run their own membase server locally we will still need to have some sort of data format to keep the data in. I'll admit not researched deeply into these technologies so they may work without a server and if that is the case then it would come down to if they are one fast and two capable of allowing retrieval of individual pieces of data inside the patch record not just the entire record itself.
    “little expense has been spared to give the impression that no expense has been spared.” - Douglas Adams

  3. #3
    Join Date
    Jan 2010
    Location
    Buenos Aires, Argentina
    Posts
    148
    I thought the same as Lee. The retrieve of a single vertex in those type of db solitions is expensive.
    But they could be a good solution for the index Lee was talking in the meeting, since it would be the most hitted part of the system.

    Can I suggest this thread get moved to the programing forum?

  4. #4
    Join Date
    Dec 2002
    Location
    Virginia Beach, VA
    Posts
    861
    Moved to the programming forum.
    “little expense has been spared to give the impression that no expense has been spared.” - Douglas Adams

  5. #5
    Join Date
    Apr 2011
    Posts
    11
    Quote Originally Posted by chronos78 View Post
    The problem with these types of solutions is that while they may work for the server side, how does the client cache their local copy of the data? Unless you propose to have them run their own membase server locally we will still need to have some sort of data format to keep the data in. I'll admit not researched deeply into these technologies so they may work without a server and if that is the case then it would come down to if they are one fast and two capable of allowing retrieval of individual pieces of data inside the patch record not just the entire record itself.
    Ahh, ok. I was only thinking of membase as a server-side solution, but I hadn't considered the fact that you don't have to read in an entire patch file's worth of data to get to a single vertex; you can just seek to the byte location in the index, read 128 bytes, and be done.

    With membase, you'd either have to read in the entire patch to access the vertex, or you'd need to store vertexes instead of patches, which would exponentially increase the size of the keyspace (possibly to the point of not working.)

    Interesting.

  6. #6
    Join Date
    Nov 2009
    Location
    Dallas, Texas
    Posts
    170
    Our studio just got bought by Zynga about 6 months ago. I know there are lots of articles out touting that farmville is running on membase, however, nothing at zynga is yet running on this as far as we've been told. We're still all Mysql and in the process of moving towords this solution. At least thats my understanding in terms of the meetings i've been in. I know my studio is looking at membase for our game, which is not Farmville, so I imagine that all zynga games will soon be using membase/couchdb. Also, while membase is only a key/value pair type of system, it is only half of the picture. The membase folks merged with couchdb so membase is used for caching while couchdb is the datastorage component. I think membase can use any persistance framework, however, if you don't choose one, the default storage mechanism is flat files. My understanding is that that Zynga will be using membase in front of couchdb, so we'll be using a full blown nosql solution. Again, all of this is pretty new so I it might change.

    One thing that is slightly different than business applications, when it comes to data concerns, is the concept of run time data vs proto data. Proto data is all the static data in the game. So proto data for a long sword would be it's name and weight, for example. The runtime data is created when the sword is assigned to a player. An example of runtime data for the long sword might be it's durability, since durability is player specific. So when you pick up a sword, you typically create the runtime version of the sword that you assign to the player from the proto data. However, you dont want to store the proto data with the player because if a designer changes the weight of the long sword you have to go to every player and update the database. This is error prone. Rather, you store only run time data with user and look up proto data in memory as required.

    This storage mechanism works best when the player is just one big JSON style document that you load into memory when the player logs in. Trying to treat the player like you would in a typical database application where you update and query the database in real time is a really bad idea. You don't want to be querying user data at run time nor do you want to be writing user data every time it changes. Zynga moves a Petabyte of data a day so control over how data is stored and moved is critical. In our case, we load the user blob into memory and then have a process that updates the player every 30 to 60 mins or so if the player data is dirty (modified). Quite a bit of server based game development revolves around finding ways to data drive everything so designers have control of the game without requiring work from programmers, which means understanding how to separate proto data (static data) from run time data (user instance data)

    Switching gears a bit, in terms of the client for the MMO, I'm not certain of the game design feature that requires terrain to be generated in real time from a db while people are playing the game or the need for a db on the client at all. The way terrain is typically done, at least in my experience in the industry, is that you use some kind of studio built tools to build the terrain and then save patches, as mesh data, to a custom file format. That includes normals, tangents, uvs, and any other calculated data so you don't have to calculate it at run time. When the user installs the game the terrain's mesh files are installed on the client machine and then streamed in from disk at run time. That way instead of worrying about vertice caches, normal caches, tangent caches, etc, you simply cache terrain meshes and stream in new ones as necessary.

    I've never heard of a technique where terrain is literally generated from vertices in a db while the game is running. Again, the typical method is to pre generate and stream in the terrain from disk. Changes to terrain are then handled by some kind of patching system that downloads new terrain and other assets to the client machine as required. There may be a requirement for the MMO that needs this, which i'm not aware of, however, for most common cases, I wouldn't recommend this approach. I would prefer streaming in files from disk as it's going to be far simpler and perform better than streaming in vertices to generate terrain on the fly. With this model, using a NoSql database on the server is perfectly fine as you really dont' need one on the client at all. The players blob is loaded into memory from the server DB and all game proto data is usually just xml (could be server db too) that's downloaded from the server and loaded into memory when the user logs in. No database required on the client.

    Even with the network based concept for editing terrain, I wouldn't use a db on the client. Using a database on the server to persist terrain for a shared, network based editing tool is fine. However, for the game, I would still export this data, as a mesh, into a custom file format that is downloaded to the client machine via a patching system and then streamed in from disk as required. I definitely wouldn't stream in vertices from a db on the client and try to generate terrain on the fly if that's what we're talking about doing.

    For the record, this is just my personal opinion based off of my experience in the games industry. I'm definitely not trying to tell anyone "they're doing it wrong". If it works it works, but i'm skeptical about a plan that involves streaming in vertices from a db and calculating normals, uvs, tangents, textures and everything else at runtime. It seems like the best way to do this would be the more traditional approach which, would be to stream in pre-generated mesh data from disk, just my 2 cents for what it's worth
    Last edited by jjguzzardo; 04-30-2011 at 02:55 AM.

  7. #7
    Join Date
    Jan 2010
    Location
    Buenos Aires, Argentina
    Posts
    148
    The thing, jj, is that we are talking about a huge terrain. Leewas talkig about 208gigs per zone. Thats something we cant store on the client.

  8. #8
    Join Date
    Nov 2009
    Location
    Dallas, Texas
    Posts
    170
    The thing, jj, is that we are talking about a huge terrain. Leewas talkig about 208gigs per zone. Thats something we cant store on the client.
    You also can't stream that kind of data over the wire in real time, especially with any kind of player base in the thousands. I said I wouldn't say "you're doing it wrong", however, if you have 208 gigs per zone than you're doing something wrong, sorry. That means you're literally streaming in vertices for the entire game, in addition to everything else the server is doing. Any type of lag on the client connection is going to cause all kinds of weird errors with people falling through the world or assets floating around or falling through the world. The traditional approach of streaming in mesh data from the client is the traditional approach for a very good reason. It's the most stable way to do this. WoW has more terrain than you would ever need for a game. Their game world is HUGE and their entire world is no where near 208 gigabytes. WoW uses the approach I am suggesting so there are ways to make huge worlds without gigabytes of vertex bloat.

    Here's just one example of a potential game play issue. Think about how long it takes to download 500 megabytes with a 12 megabit connection. Probably several minutes. Now imagine you want to fly or run from one end of the zone to the other. In order to make the world look seamless you're going to have to stream in all the vertices for the world along the way. If a zone is 208 gigabytes I'm going to get on my flying mount, wedge the w key down, and go to sleep because it's going to take hours to stream in all the vertices to make the world seem seamless while I'm flying over it. In fact, if a single zone is 208 gigs it would probably take days to stream in all that data. This is just one of many issues you're going to run into from a game play point of view, not even considering the size of the server farm required to do this for a medium to large size MMO.

    There's good reasons games don't use this approach so again, I think sticking to the traditional method is best and if you find you're world is terabytes in size, than you need to to find a different approach. I'm pretty sure there's a misunderstanding here somewhere, I'm just discussing the concept of trying to stream in worlds that are terabytes in size in real time. If this is the case I really suggest going back to the drawing board. Again, I'm not trying to be a jerk or rain on anyone's parade, I just really think this would be a bad idea if this approach were the one chosen for terrain. No offense is intended. Again, I'm pretty sure there's a misunderstanding here somewhere because a world terabytes in size doesn't make sense.
    Last edited by jjguzzardo; 04-30-2011 at 01:53 PM.

  9. #9
    Join Date
    Jan 2010
    Location
    Buenos Aires, Argentina
    Posts
    148
    You should watch meeting n7. The world is like 1000 times bigger than wow. and the idea is to stream the patches as needed. There is also 3 dif resolution of patches. But again watch meeting 7.

  10. #10
    Join Date
    Dec 2002
    Location
    Virginia Beach, VA
    Posts
    861
    JJ the only reason I believe that this will be possible is because the world is so large. Early estimates put the amount of data per patch that has to be streamed to the client is about 3-70k depending on compression and amount of modifications to that patch. Which isn't too bad considering. Now when you put that into context of the size of a zone and travel speeds something interesting becomes apparent. Let's say walk speed is 1.5m/sec and since no one walks in MMO's run speed will be our low end travel speed at 3m/sec. At this speed it takes 7.4hrs to cross a zone end to end. If we say that are max flight speed is 800% or 24m/sec it still takes almost and hour to cross. If one were so inclined to explore the entire zone at flight speed it would take them 31 days to see every patch in a zone if they never once stopped nor crossed a patch more than once. Even the slowest internet connections should be able to keep up at that speed.

    Now this would be far from you typical play experience. More likely a player would find themselves playing in only parts of the zone depending on their current status in the game. These parts could easily be cached on the client side so that once downloaded from the server they would only need to be occasionally checked to see if there wasn't newer versions available on the server. The client side cache would also have a purging feature that would unload the least frequently patches from storage if the cache starts to reach a user defined limit. This way we allow the player to decide how much of their hard drive they wish to allocate to the game. It also prevents us from having to send 100's of gigabytes of data that would have to be stored on the client. Something else to consider is that the data storage requirements on the server are different than that of the client since the client only needs height, visibility, and texture splats while the server requires a lot more to keep track of all that it need to simulate the game world. This means where the server may need around 208GB a zone the client only need about 45GB. Of course we won't expect the client to save all the data for the zone at once, just the parts they need for the areas that they are currently playing in. Personally I thought this was a rather elegant solution for a fairly complex problem but we won't know for sure if it will work until we get a demo up and running to test it.
    “little expense has been spared to give the impression that no expense has been spared.” - Douglas Adams

Page 1 of 5 123 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •