Results 1 to 10 of 10
  1. #1
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285

    Question about Nelson's serialization system

    Hi
    I watched up to the rune slinger (chapter 8) episode 4 of MMO, and nelson in episode 3 mentions that he is working on a serialization system, so here is some questions I have :
    1- did he teach it until the end of chapter 11 or not yet?
    2- is that system capable of serializing something like Dictionary <string, int>, how about Dictionary <string, List<T>> ?
    3- he said his system are close to what protobuff has, correct me if I am wrong but for protobuf, when you need AOT development, you have to take extra step and make an exe file that will compile a dll, that dll will tell the system what should be serialized, so basically annotating members inside the main project would be separated to another project and that is annoying. with IL2CPP ahead of us, question is, will nelson's system need such a thing or would annotating members and classes like the way unity itself is doing work properly?
    4- can his system serialize/deserialize Unity.Object ?

    thanks
    Last edited by css_maya; 09-09-2015 at 10:34 AM.
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

  2. #2
    Join Date
    Feb 2014
    Posts
    277
    Runeslinger was quite a while ago and it was a lot of fun except when it ended with all the loose ends.

    But your questions are all related to chapter 11. My memory doesn't let me answer your questions in detail, but here are some thoughts:

    In amongst the unit testing is a fairly simple serializer/deserializer that covers tons of types. The list grew video by video as Nelson added more and more (see the 7 videos with Component in the title). I can't remember all or many of the different types of things that can be serialized, and a quick look back at the first half dozen videos about the component map didn't locate a concise list.

    The latter half of chapter 11 is all about working with the new Unity 5, IL2CPP, and Nelson's search for a way to make it work and still pass all the unit tests. In there are some fascinating videos on code generation and Roslyn. I wouldn't be too surprised if Nelson didn't re-hash it again though after it gets some game testing.

    Chapter 11 is long, but I really enjoyed doing it twice. And I will probably do it again when I cut over to Windows 10 and Visual Studio 2015. It appears that Nelson is already using VS2015 but my early attempts broke a few existing projects so I will have to redo it in more detail.

    You can't just code in chapter 11 as it builds heavily on chapter 10. In other words, I think that the creation of the MMO actually began in chapter 10 with things that Nelson was already experienced with ie web frontend and Continuous Integration.

    Progress is slow, but Nelson is but one person.

  3. #3
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285
    thanks for reply, about the slow progress, i understand it, but at least, i would think nelson would kick it off faster than before with their new announcement, hence the reason i gone for 1 year sponsorship, also, just knowing that he had implemented it in front of camera is enough for me, i would never jump a chapter because i learn stuff in EACH video and when i hit the point for serialization i will see if he covered those few types or not, and if not, god help nelson with me spamming his pm inbox until i get a reply from him
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

  4. #4
    Join Date
    Aug 2010
    Posts
    158
    So i just added a test to see if that is possible and it worked like a charm.

    I created a Dictionary<string, List<Component1>> and while I am not testing generics, if you are using T then you are passing in a Type and should be the same.

    Also note that I am not testing it with multiple items in the dictionary or the list. That would be a simple addition though


    Code:
    public void CanReadAndWriteDictionaryOfStringAndList()
            {
                var ser = new SimpleSerializer();
    
                var key = "a";
                var component1 = new Component1
                {
                    Float1 = 3.14f,
                    Int1 = 256,
                    StringArray = new[] { "Hello", "World", null, "BLEH" }
                };
    
                var objList = new List<Component1>();
                objList.Add(component1);
    
                var dict = new Dictionary<string, List<Component1>>();
                dict.Add(key, objList);
                
    
                this.TestSerializer(
                    bw => ser.WriteObject(bw, typeof(Dictionary<string, List<Component1>>), dict),
                    br =>
                    {
                        var readComponent = (Dictionary<string, List<Component1>>)ser.ReadObject(br, typeof(Dictionary<string, List<Component1>>));
                        readComponent.Should().NotBeNull();
    
                        readComponent.ContainsKey(key).Should().BeTrue();
    
                        readComponent[key].Should().NotBeNull();
                        readComponent[key].Count.Should().BeGreaterThan(0);
    
                        readComponent[key][0].Should().NotBeNull();
                        readComponent[key][0].Int1.Should().Be(component1.Int1);
                        readComponent[key][0].Float1.Should().Be(component1.Float1);
    
                        readComponent[key][0].StringArray.Length.Should().Be(component1.StringArray.Length);
                        for (int i = 0; i < readComponent[key][0].StringArray.Length; i++)
                            readComponent[key][0].StringArray[i].Should().Be(component1.StringArray[i]);
                    });
            }

    Nelson covers *basic types*. All of your complex types are going to be derived from basic types and therefore should be covered.

    ** for now this includes uint, byte, string, float, double, decimal, int, short, ushort, long, ulong, char, bool, Guid, DateTime
    Last edited by am385; 09-10-2015 at 04:59 PM.

  5. #5
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285
    i see, so something complex like animation curves won't get serialized by themselves, but i have a helper method that converts them to byte[] and back, and that should work ha? by the way, i have a noob question about serialization, how can we make sure something is really serialized within unity without using unit tests? for example when i am using unity's build in serializer for something, how can find if the stuff got serialized properly or not? also another question but i didn't want to make a new thread for it to keep forum clean, i went to photon server's website and they are joining forces with bolt, and the whole website was deferent, it rose the question in my mind, what will nelson use for future tutorials to handle networking? new built in unity system, any special version of photon, or bolt now?
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

  6. #6
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285
    OUCH, just watched first minutes of his simple binary serializer, NOT for persistent data? and not the whole solution he was showing previously? DAMN DAMN, not another not completed part, seriously Nelson, how many loose ends you guys left during these series.
    I would want to see a whole serialization series that is not ONLY good for serializing data during the net but also for saving data on hard drive.
    DAMN
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

  7. #7
    Join Date
    Aug 2010
    Posts
    158
    As far as serialization goes, the simple serializer is all we need for getting efficient data moving over the network. If you are looking for persistent state for large state of data, I would recommend a Object/Document DB or another NoSQL based database. They usually save everything off as JSON/BSON and you can directly serialize a complex object to them and save it with a key. You do have the option of writing our serializer and writing the data directly to file but that would be more difficult to track as you would need some way to reference each set of data.

    Riak, mongo, raven, Azure DocumentDB, ... there are lots.


    You could also use an actor framework as a middleman with data to have better performance with an in memory solution and the ability to queue data persistence.

    That being said though, I think the SimpleSerializer is great for now. It has everything we need to get things working and have increased network performance.

  8. #8
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285
    I think I rephrased myself bad, look, when you hear serialization, and when you hear that nelson is working on a serialization and he compares that system to something like protobuff, you would think that it would be a complete solution that covers everything, not only data streaming. also, I don't think using databases is a nice idea here, think about an asset that you are developing for other unity users to sell it on asset store, if you use any special database, that would force every user of your system to use that data base regardless of they feeling of liking it or not, there is a big potential there that you will loose some customers just for this simple fact. also I agree that json and bson are good but to be honest, when I saw the chart on protobuff's benchmark, I will never ever go back to those two solutions. the only reason I would like to see something other than protobuff, is how protobuff handles serialization for AOT, for that you would need to sit and write an executable file that had reference to every class you are developing and that is a big handicap in my opinion, I would really like to see nelson's solution in the way that it would eliminate the need of such a thing. it would be easy to tweak the system in a way that it use a simple Boolean flag on an attribute for data stream on net, so when it was true, you would go the way it is implemented right now if false, it would go for file serialization with the needed stuff.

    please nelson, we need something like this sooner or later in our projects, and I hope that won't take long to make.
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

  9. #9
    Join Date
    Aug 2010
    Posts
    158
    Ahh, i get what you mean. I do like that Nelson's serializer is similar to protobuf in that we are sending compressed binary data. Instead of constructing a schema we are parsing the classes and creating the generated files at compile time. Very similar. I was going to say if you wrap his current serializer needs a binary writer. You could very easily create binary writer with a fileStream instead of a memory stream and accomplish something similar.

  10. #10
    Join Date
    Jun 2008
    Location
    Iran / Tehran
    Posts
    285
    I had thought about the same idea, problem is, I think he is dropping every type definition? that would be a big problem
    If you want to learn something, start teaching about it to others, nothing will help you more than that !

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •