0

We generate reports in our web application by querying our sql server for data returned as xml and then processing with xslt to create the final output. As a way to speed up the system, we removed all the static information from the returned sql xml and cached a large XDocument with all the static info in it. Right before performing the xsl transform, we append the XDocument with the static info to the end of the xml that came from sql server. The static XDocument is about 50Meg and takes many seconds to build from the sql server.

Our problem is that once we started caching a few of these large XDoc's, we hit the cache private bytes limit size and the cache was cleared. Rebuilding these XDocuments is too time consuming to do while people are running reports. I have not tried saving these XDocs to a physical file because they are needed for every report run which happens constantly through the day.

I've thought of installing AppFabric Cache but I'm not sure it's a great idea to store 5 to 10 of these large items in it.

Any ideas? If I install more memory on the web server, will it automatically be available to asp.net for a larger cache? I've tried compressing the data before storing it in the cache (shrunk by a factor of 5), but uncompressing it and reparsing the XDocument slowed the server way down.

4
  • Why not use a simple custom cache on database / or save them as files on disk (better) and give this files to download from the users using a handler.
    – Aristos
    Mar 24, 2012 at 0:32
  • @Aristos The user doesn't download the static data, it's combined with the xml that comes from the sql server, then processed through xslt to create the report output. I can save the data on disk, but I then need to load it back via XDocument.Load() for every report. Thought there must be a faster way.
    – Wavel
    Mar 24, 2012 at 18:02
  • 1
    save to the disk the serialize version of the final XDocument.Load() using a real fast serialize class like the code.google.com/p/protobuf-net
    – Aristos
    Mar 24, 2012 at 18:07
  • @Aristos If you repost your comment as an answer, I can mark it as the answer I'm going to use!
    – Wavel
    Mar 25, 2012 at 20:21

2 Answers 2

0

Final, just save it to a file as it is and then reload it as it is because its all ready Serialized.

The protobuf-net is super fast and light and I have test it and use it, but it not make any good because its all ready serialized.

2
  • Just heard back from Marc Gravell who states that protobuf-net is not a good solution for serializing XML since it will just store it as a string. Basically, no better than XDocument.Save().
    – Wavel
    Mar 26, 2012 at 19:52
  • @Wavel oh, you have right, its all ready XML, I didnt think of.... hmmm I change the answer.
    – Aristos
    Mar 26, 2012 at 21:03
0

You can serialize the xml object in a binary format and store it on the database using a varbinary(max). Not sure about the performance of that, but It might worth to try it since it won't take very long to implement it.

Something else that you might want to address is the performance penalty for the first user accessing the report. In order to avoid this, you could pre-generate the reports so they are cached for everyone.

2
  • If I serialize the xml onto the sql server, then I'm kind of back to square one where the xml returned has both they user's data and the static data in one large xml. I can't pre-generate the reports as they are ad-hoc reports that are designed by the user. The user data is also constantly changing.
    – Wavel
    Mar 24, 2012 at 18:05
  • I got you wrong, I thought the report kind of limited and everyone was getting the same and that's why you wanted to use caching Mar 24, 2012 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.