1

Infrastructure and preamble

I have a PlayFramework (2.3.8) App hosted on an AWS EC2 instance. I have an array of complex objects, which should be returned as a JSON string via a web API. I need a deep copy of the array, with all child objects fully loaded until the very last layer. The array has the size of 30-100 entries, each entry has around 1-10 entries, each entry of those has up to 100 properties, in the end there are no BLOBs or similar involved, it all boils down to strings/doubles/ints/bools. I am unsure how far the exact data structure is of importance, please let me know if you need more details. The resulting .json file size is about 1 MB.

The performance of deserializing this array is awful, for the ~1 MB on my local machine it takes 3-5 minutes; on the EC2 it takes about 20-30 seconds.

The initial problem: poor performance when using play.libs json

My array of objects is loaded and stored as a JsonNode. This JsonNode is then forwarded to an ObjectMapper, which finally writes it prettyPrinted:

List<myObject> myObjects = myObjectService.getInstance().getAllObjects(); // simplified example

JsonNode myJsonNode = Json.toJson(myObjects); // this line of code takes a huge amount of time!

ObjectMapper om = new ObjectMapper();
return om.writerWithDefaultPrettyPrinter().writeValueAsString(myJsonNode); // this runs in <10 ms

So I nailed down the culprit to be the Json.toJson deserialization. As far as I could find out, it is a sort-of-wrapped Jackson library which is used by the PlayFramework.

While I have read about some performance issues of JSON deserializing, I am unsure if we should be talking about some hundred-milliseconds to seconds, and not minutes. Anyway, I tried implementing some other JSON libraries (GSON, argonaut, flexjson), which didn't really go smoothly.

GSON

I "simply" tried replacing the play-json library with the GSON library, as I did on another small part of the project. It worked fine there, but even though I have NO circular references, it throws StackOverflowErrors at my face, even if I try to deserialize a tiny manually created object. Both on my dev machine as well as on the EC2 instance.

FlexJson

List<myObject> myObjects = myObjectService.getInstance().getAllObjects(); // simplified example

JSONSerializer serializer = new JSONSerializer().prettyPrint(true);

return serializer.deepSerialize(myObjects); // returns a prettyPrinted String

Worked quite okay so far, it takes only around 20% of the time compared to the Json.toJson method above. Which could be, however, because it doesn't REALLY deep copy the objects. It does deep copy it on the first layer, however since my model has some more complex properties (with childs and grandchilds and grandgrandchilds...), and quite a lot of them, I am unsure how to procede here.

Here is the example output of one of my nested objects (this is one of the properties of the "upper" object):

 "class": "com.avaje.ebean.common.BeanList",
                "empty": false,
                "filterMany": null,
                "finishedFetch": true,
                "loaderIndex": 0,
                "modifyAdditions": null,
                "modifyListenMode": "NONE",
                "modifyRemovals": null,
                "populated": true,
                "propertyName": "elements",
                "readOnly": false,
                "reference": false

Do you have any other solution suggestions, or hints what might be broken? I was also thinking about that maybe the entities are only FULLY loaded once I call .toJson()? Still it shouldn't take such an amount of time.

Thanks in advance!

3
  • 1
    It is possible that "the entities are only FULLY loaded once I call .toJson()". As I see, you are using Ebean. Try activate logging for sql statements to see if the object graph is being loaded before or during the toJson call. Nov 9, 2016 at 20:05
  • Also you can try to use something like VisualVM or YourKit to find out what is making it slow, or save the generated json file. load it to memory (parsing it) and then rendering it with toJson (if this process is quicker than the previous then clearly something other than json parsing/generation is making it slower)
    – Salem
    Nov 9, 2016 at 23:07
  • Thanks for the tip with the sql logging, good idea. Turns out that the entities and their child-entities are only loaded during Json.toJson(). I tried changing fetch = FetchType.Eager for those child-entities, but this made no difference at all. And basically I don't really see what difference it would make, at what point the entities are loaded, so why not during the output for .toJson(). But it seems like the bottleneck is with the entities/DB, not with the JSON serialization..?
    – konrad_pe
    Nov 9, 2016 at 23:39

1 Answer 1

0

TLDR: this issue had nothing to do with PlayFrameworks JSON deserializing performance, rather than with some eBean / database issues. Enabling SQL logging in application.conf pointed me to this.


Further remarks and thoughts: Thanks to the hint of marcospereira in the comments, I nailed the problem down to be a fetch issue within play / ebeans, rather than a JSON performance issue.

Obviously my entities are loaded lazy (/flat) at first, by enabling SQL logging I could see that the correct prepared SELECTs are only fired once my code hits .toJson(). So many of the child objects are only fetched from the database when calling .toJson(), which results in a couple of hundred SELECTs and therefore quite some time to finish.

Playing a bit with the RDS instance scales I found some very weird results. This isn't REALLY related to the question initially asked, yet I want to share my findings, maybe it can be of help for somebody out there. Read about it in the section below.

RDS scaling experimenting...

In my dev environment (t1.micro) I hooked up a copied instance of my prod DB on a small RDS instance (db.t2.micro), to see if anything changes.

My prod environment (t2.large) + prod RDS (db.t2.large) took around 19.5s to finish the API call. The NEW dev environment (t1.micro + db.t2.micro), which is weaker on both computing as well as db, took only about 10.5s, which is highly inconclusive, as basically both instances ran the very same code, only pointing to another DB server (with identical db content). I switched the dev DB to db.m4.large to see if that brought any improvement, and the load time went down to about 5.5s.

I am completely puzzled why the faster prod EC2 instance would require more time for the exactly same API call than the dev instance. In the end I changed my prod db class from db.t2.large to db.m4.large and have a response time of 4.0s now.

Feels like the "old" prod DB instance was sort of worn-out/clogged (is there such a thing? I somehow doubt it...). Even the smaller dev instance + dev db responded much quicker. Even though the different RDS scalings brought some improvement, I doubt that the difference between db.t2.large -> db.m4.large would cause a change in that magnitude.

Maybe if someone has some ideas what's going on, I would be very happy to discuss this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.