Announcement

Collapse
No announcement yet.

[FIXED] [#3915] Newtonsoft.Json.JsonConvert.DeserializeObject() is slow

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [FIXED] [#3915] Newtonsoft.Json.JsonConvert.DeserializeObject() is slow

    Hello,

    I'm using Bridge.net in one project, which went live last year. Now the data start getting bigger and I see, that Newtonsoft.Json.JsonConvert.DeserializeObject is taking quite some time on the JavaScript side. I traced it and inside the function the native JSON.parse is quite fast, but then, converting the result of JSON.parse to the ".net" object takes the time. In my case its adding up to around 5-15 seconds right now depending on the speed of the CPU.

    Is there a way to just use JSON.parse and then somehow cast the object to the .net object?

    Thanks,
    Markus

    #2
    Hi. Can you demonstrate as much of the scenario you would like to see working using a https://deck.net sample?

    Comment


      #3
      Hello geoffrey, I try to create a small sample until tonight...

      Comment


        #4
        Hello Geoffrey, here is a Deck: https://deck.net/3c701c4418c2bef81629458b7d75647d I don't get the jQuery Ajax Call to load the content of the file. MayBe this is a cross Domain issue...??? You can download the file with the browser and than execute the Deck Code locally...

        The Code showing the problem is:
        // Deserialize the json back into a real .Net Object
        DbElementCommandTO cmdTO = JsonConvert.DeserializeObject<DbElementCommandTO>(data.ToString());

        Comment


          #5
          Hello @Markus!

          When I was able to run the code in a local Bridge project, running a local iisexpress server to "wrap" the data.json file, Bridge wasn't able to deserialize its contents, it seems the class prototype you wrapped in the demo.

          If that's just related to performance of newtonsoft.json parsing, can't you wrap up the object adding some random data (to the quantity that affects you), and just serialize() then deserialize(), just to reproduce the issue the way you see it? Well, that is, if that's just performance in the deserialization process the report's about.

          Comment


            #6
            Hello Fabricio,

            here is a little Bridge.Net Test Solution which generates random Data, serializes it and calls the deserialize:
            https://www.mahop.net/files/BridgeSerializeTester.zip

            Nice greetings, Markus

            Comment


              #7
              Hello Fabricio,

              did the sample project help you to reproduce the issue? And if yes, do you think, that there will be a solution?

              Kind regards,
              Markus

              Comment


                #8
                Hello Markus!

                Sorry I didn't have time to check this earlier. Opening up and running a project takes much more effort than just checking a deck. By the way, I just pasted your `App.cs` in Deck and I can run the example just fine, have you tried that?

                https://deck.net/694b39cd17dfc09e580483672c8be988

                So I guess we won't need the project anymore.

                Furthermore, as retyped was used just for the performance counts, I could simplify this for the following Deck:
                https://deck.net/b672e0b116e7d773b6d0a942535726d3

                using Bridge;
                using Newtonsoft.Json;
                using System;
                using System.Collections.Generic;
                
                namespace BridgeSerializeTester.Lib {
                    public class App {
                        public static void Main() {
                            GetData();
                        }
                
                        public static void GetData() {
                            // Configure the Ajax request
                            var tStart = DateTime.Now;
                            Console.WriteLine("Starting...");
                
                            var dbElementCommandTO = new DbElementCommandTO();
                            dbElementCommandTO.DBElements = new List<DbElementTO>();
                            dbElementCommandTO.DBElementValues = new List<DbElementValueTO>();
                            dbElementCommandTO.DBElementLinks = new List<DbElementLinkTO>();
                
                            var now = DateTime.Now;
                            for (var i = 0; i < 1000; i++) {
                                var elementID = Guid.NewGuid();
                                dbElementCommandTO.DBElements.Add(new DbElementTO{ElementID = elementID,ElementTypeKey = 10,NamespaceKey = 1, TimeChanged = now, TimeCreated = now, TimeChangedMaster = now, UserChangedID = Guid.NewGuid(), UserCreatedID = Guid.NewGuid()});
                                for (var j = 0; j < 10; j++) {
                                    dbElementCommandTO.DBElementValues.Add(new DbElementValueTO{ElementID = elementID, ValueKey = j, Index = 0, LinkID = null, ValueType = 1, ValueString = "Data to Store " + j + i});
                                }
                                for (var j = 0; j < 10; j++) {
                                    dbElementCommandTO.DBElementLinks.Add(new DbElementLinkTO{ElementID = elementID, Index = 0, LinkID = Guid.NewGuid(), LinkElementID = Guid.NewGuid(), TimeChanged = now, NamespaceKey = 0});
                                }
                            }
                            Console.WriteLine("Generate Data took " + (DateTime.Now - tStart).TotalMilliseconds + " ms");
                
                            tStart = DateTime.Now;
                            var jsonStr = JsonConvert.SerializeObject(dbElementCommandTO);
                            Console.WriteLine("Serialize Data took " + (DateTime.Now - tStart).TotalMilliseconds + " ms");
                
                            // Deserialize the json back into a real .Net Object
                            tStart = DateTime.Now;
                            var cmdTO = JsonConvert.DeserializeObject<DbElementCommandTO>(jsonStr);
                            Console.WriteLine("Deserialize Data took " + (DateTime.Now - tStart).TotalMilliseconds + " ms");
                
                            Console.WriteLine("Elements: " + cmdTO.DBElements.Count);
                            Console.WriteLine("ElementValues: " + cmdTO.DBElementValues.Count);
                            Console.WriteLine("ElementLinks: " + cmdTO.DBElementLinks.Count);
                
                            tStart = DateTime.Now;
                            //@ var jsonStr2 = JSON.stringify(dbElementCommandTO);
                            Console.WriteLine("JSON Serialize Data took " + (DateTime.Now - tStart).TotalMilliseconds + " ms");
                
                            // Deserialize the json back into a real .Net Object
                            tStart = DateTime.Now;
                            //@ var cmdTO2 = JSON.parse(jsonStr2);
                            Console.WriteLine("JSON Deserialize Data took " + (DateTime.Now - tStart).TotalMilliseconds + " ms");
                
                            //@ System.Console.WriteLine("Elements: " + cmdTO2.DBElements.length);
                            //@ System.Console.WriteLine("ElementValues: " + cmdTO2.DBElementValues.length);
                            //@ System.Console.WriteLine("ElementLinks: " + cmdTO2.DBElementLinks.length);
                
                        }
                    }
                
                    public class DbElementCommandTO {
                        public string AuthToken { get; set; }
                        public string Key { get; set; }
                
                        public string Cmd { get; set; }
                
                        public bool Success { get; set; }
                        public string ErrorMsg { get; set; }
                        public int ErrorNo { get; set; }
                
                        public List<DbElementTO> DBElements { get; set; }
                        public List<DbElementValueTO> DBElementValues { get; set; }
                        public List<DbElementLinkTO> DBElementLinks { get; set; }
                    }
                
                    public class DbElementTO {
                        public short NamespaceKey { get; set; }
                        public Guid ElementID { get; set; }
                        public int ElementTypeKey { get; set; }
                
                        public bool IsReadOnly { get; set; }
                
                        public DateTime TimeChanged { get; set; }
                        public DateTime? TimeChangedMaster { get; set; }
                        public DateTime TimeCreated { get; set; }
                        public Guid UserChangedID { get; set; }
                        public Guid UserCreatedID { get; set; }
                
                        public sbyte DeletedStatus { get; set; }
                        public int Version { get; set; }
                    }
                
                    public class DbElementValueTO {
                        //Keys
                        public short NamespaceKey { get; set; }
                        public Guid ElementID { get; set; }
                        public int ValueKey { get; set; }
                        public Guid? LinkID { get; set; }
                        public int Index { get; set; }
                
                        //Values
                        public int ValueType { get; set; }
                        public string ValueString { get; set; }
                        public long? ValueInt { get; set; }
                        public decimal? ValueDecimal { get; set; }
                        public DateTime? ValueDateTime { get; set; }
                        public Guid? ValueGuid { get; set; }
                        public string ValueArrayBase64 { get; set; }
                
                        public DateTime TimeChanged { get; set; }
                    }
                
                    public class DbElementLinkTO {
                        //Keys
                        public short NamespaceKey { get; set; }
                        public Guid ElementID { get; set; }
                        public Guid LinkID { get; set; }
                        public int Index { get; set; }
                
                        public Guid LinkElementID { get; set; }
                
                        public sbyte DeletedStatus { get; set; }
                        public DateTime TimeChanged { get; set; }
                    }
                }
                And then run it in native C# with this dotnet fiddle.

                This way it is much easier for us to compare the code and performance. I've created issue bridgedotnet/Bridge.Newtonsoft.Json#151 to track this performance issue, so we post a notification here whenever we do some performance improvements to the deserialize code.

                Unfortunately, there's just so much it is done in terms of sanity checks and normalization when serializing and deserializing that I don't know an easy way to just use the native JSON.parse() and have it map nicely into a C#-bound object. What you could use though, is a simple object (or ObjectLiteral object), and do the mapping in C# side. But the parsing from string to Guid, DateTime, and so on may make it as slow (or even slower!) than it actually is using Bridge.Newtonsoft.Json implementation.

                Sorry this doesn't really help your issue, but I know no quick solution at hand.

                Comment


                  #9
                  Hello Fabricio,

                  thank you very much for the answer! (And also for the Links. I like dotnetfiddle.net. I did not know about this site...)

                  So I will work on my code and on the amount of data loaded per step to speed up things again...

                  Kind regards and nice greetings,
                  Markus

                  Comment


                    #10
                    Hello Fabricio,

                    I debugged your code a little and found out, that around 80% of the time is spend in System.Guid.parse() and System.DateTime.parseExact() if I replace this two calls in the DeserializeObject function and replace them just with return null the code is around 5 times faster... And most time is spend in System.Guid.parse().

                    I don't know much about your JSON serializer and if the values of this two types are always in the same format, but if yes, everything could be a lot faster if there are specialized functions for creating a DateTime and a Guid from the JSON values which are faster than the parse and parseExact functions.

                    Kind regards,
                    Markus
                    Last edited by geoffrey.mcgill; 2019-04-11 @ 05:05 PM.

                    Comment


                      #11
                      We have added pull request with a fix which improves GUID parsing time. If use Fabricio test case then time was reduced from 5570 ms to 3089 ms. Still not good enough but already better.

                      Comment


                        #12
                        Hello Markus!

                        The pull request looks good and should go to the next Bridge release, 17.8.0 currently.

                        On my side, it turned out that the serialization speed (which was not subject to changes here) is now about the same for the deserialization (which has been optimized), which is about 2x faster, so better now.

                        Thanks again for reporting the performance issue!

                        Comment


                          #13
                          Hello Fabricio,

                          thanks for the info and the help! I will upgrade as soon as possible. Do you have a date for the release?

                          Kind regards, Markus

                          Comment


                            #14
                            Hello Markus!

                            We don't have one. We're reviewing the issues and still fixing some others that will still hopefully go to 17.8.0. If you use twitter, follow us to ensure you receive a notification as soon as it is released.

                            Comment

                            Working...
                            X