Search

So I've been working on a project where I need to serve up a large data set in JSON. The data set is around 3500 entries each with six fields.

I can serve up to around 1000 before it fails but even at this many entries the load time is ridiculously long. This wouldn't be a problem if I could get the whole record set to load as the Cachelite extension does a great job at providing a more than acceptable time frame after the first load.

I've profiled the page (image attached) and it all looks fine to me except for the XSLT Transformation is at 96s!

Hoping someone can help me with how I can go about addressing this issue.

Thanks in advance for your help!

btw I've updated to 2.2 (nice work to all the contributors) and I'm running on a Amazon EC2 micro instance which has 613M of memory

Just to count out hardware I've booted a instance with more power:

High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of local instance storage, 32-bit platform

Significant improvement at 16s compared to 96s but it still wont do the full data set.

Attachments:
Screen shot 2011-02-27 at 5.36.20 PM.png

I think there's little you can do.

If you have some spare time, I'd be interested to see how this modifications affect your issue. It was developed to address a similar problem in the past, so it might turn handy.

The XML is being built in 2-3 seconds, wich is "acceptable" compared to 16-100 seconds for the XSLT. Unfortunately all Symphony does is call PHP's XSLT processor and let it do the work so there's really not much one can do to improve it besides improving your XSLT. Can you post it on Pastie or Gist so we can have a look?

I am not sure if the translation from the XMLelement tree to a string is done inside the "XML built", "XML generation" or the "XSLT transformations" profiling step though. Maybe that's causing the bottleneck here.

From my experience the performance of transforming XML into JSON depends a lot from the XSLT. And the XSLT depends on your needs, e.g. if you want attributes to be transformed.

I found this utility working very well for my needs.

@alpacaaa yeah sure I'm willing to see the impact that this modification would have but am I wrong in assuming that this would have a impact on XML build and generations? It seems to me that the problems lies after the xml is generated, or am I wrong?

@phoque cheers if you could have a look that would be great. I'm still pretty new to XSLT but it is pretty basic http://pastie.org/1612805.

Note: I'm using the XML to JSON utility http://getsymphony.com/download/xslt-utilities/view/26420/ I might see what the load times are like without using this utility might help identify if this is the bottle neck.

Have you tried to output your JSON directly? Something like:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output
    method="html"
    omit-xml-declaration="yes"
    encoding="UTF-8"
    indent="no" />

<xsl:template match="data">
    {<xsl:apply-templates select="get-prices/entry" />}
</xsl:template>

<xsl:template match="get-prices/entry">
    prices: {
        id: <xsl:value-of select="@id"/>, 
        station-id: <xsl:value-of select="station/item/@id"/>, 
        value: <xsl:value-of select="price"/>, 
        fuel-type-id: <xsl:value-of select="fuel-type-id/item"/>, 
        time-stamp: <xsl:value-of select="time-stamp"/><xsl:text> </xsl:text><xsl:value-of select="time-stamp/@time"/>
    }
</xsl:template>

</xsl:stylesheet>

Ok so I've removed the XML to JSON utility and it is definitely the source of the excessive load times. Without converting to JSON I'm getting 5s total for all 3500 entries!

Much better but I'm using Symphony to serve data to an iPhone application that I'm building and JSON is much easier to work with on the iPhone than XML. Could the XML to JSON utility be optimised? Like I said I'm new to XSLT so any guidance here would be much appreciated!

@klaftertief is this what you were referring to? http://www.xml.lt/Blog/?search=xml+to+json How did you utilise this with Symphony? What size dataset where you converting and what was the process time?

Attachments:
Screen shot 2011-02-27 at 8.01.54 PM.png

BOOM... Nils = genius!!

Much more straightforward approach and loads all entries in 5s which is more than acceptable as I'll be caching the page as the data only changes once a day.

Thanks so much for you help!

Yeah that makes perfect sense. The utility runs recursively through your XML tree, testing for different types (object, array, bool etc) over and over again. Needless to say, for such a big yet flat dataset like yours it's less optimal than a single, simple template like Nils'.

Glad you were able to find the problem.

@klaftertief is this what you were referring to? http://www.xml.lt/Blog/?search=xml+to+json How did you utilise this with Symphony? What size dataset where you converting and what was the process time?

Yes. I used it as a normal utility and changed it slightly by adding a named template to it which I call in my page XSLT.

<xsl:template name="json">
    <xsl:param name="xml" />
    <xsl:apply-templates select="exsl:node-set($xml)" />
</xsl:template>

I needed a general XML-to-JSON utility to merge data from different datasources and create a list of date related data. The size of the dataset depends on the chosen timeframe. Converting two years of data (merging and transforming a set quantitive information into a list of 700 days) needs about 1.25s in the XSLT.

@Brendtos

Sounds like you've got this fixed, and you're probably past this point, but I've actually served a plist from symphony to an ios app, which I believe is very easy to handle on the ios side.

Thanks for the tips, folks! I ran up against the same bottlenecks doing JSON conversion on ~400 entries with the XML to JSON utility, and ended up scrapping that part of my project because the performance was poor. This should help with the next attempt...

Note: updated the post title to help people find this thread if they are having the same problem.

@jonasd glad this helped. It was a very frustrating problem which had such a simple solution!

@ashooner yeah never thought of that to be honest. I do like working with JSON though and there is a nice library that I work with that converts it all into appropriate objects. http://code.google.com/p/json-framework/

Thanks again to everyone for your help, it is much appreciate!

@Nils implementation seems to be cleaner but I can't get it to work. The JSON is malformed due to a missing comma right after each entry. @Brendtos, how did you manage to solve this?

FIXED: You have to dynamically check out if there is another sibling.

<xsl:if test="./following-sibling::entry">,</xsl:if>

Where entry is the node name.

Create an account or sign in to comment.

Symphony • Open Source XSLT CMS

Server Requirements

  • PHP 5.3-5.6 or 7.0-7.3
  • PHP's LibXML module, with the XSLT extension enabled (--with-xsl)
  • MySQL 5.5 or above
  • An Apache or Litespeed webserver
  • Apache's mod_rewrite module or equivalent

Compatible Hosts

Sign in

Login details