Search

I experience a special problem with one of my websites: To run a newsletter, I used to import some thousand recipients using a custom PHP script which returned an array. This output was included using a rather dirty hack. In simulations even an array with some hundred thousand recipients worked without issues (using 32 MB of local memory).

Then I decided that an extension providing a custom datasource would be a much cleaner solution. It works! Using some thousand recipients (= entries), the XML is created properly. But: XSLT transformation breaks for more than approx. 1000 recipients. Changing the local value of PHP’s memory_limit does not seem to make any difference.

Does anybody have any hints? Is this a Symphony issue or a libxslt issue?

(I tested this on Symphony 2.0.6 and 2.0.7 — no difference.)

[EDIT]: I changed the title of this thread to make it more verbose.

When the transformation breaks, do you get an error message?

On the frontend I get the standard message:

XSLT Processing Error  
This page could not be rendered due to the following XSLT processing errors.  
General  
Compile

In debug mode I see that the XSLT output is blank

The XSLT process dies without any verbose errors. This is connected to the overall number of entries, but some thousand entries shouldn’t make any memory problems. I can replicate this behavior on my local development machine. I am a bit lost.

Any ideas?

The really strange thing is: increasing the local and/or global memory_limit value for PHP makes no difference at all.

What version of PHP and what server type (Apache/Litespeed) are you running on? It could be your version of PHP has a possible memory leak. If you look over the PHP 5 changelog and search for xsl on the page, you’ll find several fixes for XSL memory leaks. That might be your culprit.

Thanks for the tip!

I have tested this issue on completely different machines with nearly the same result:

  • Debian Etch, PHP 5.2.0-8, LibXSLT 1.1.19 (PHP memory_limit global/local 16M/32M)
  • Mac OS X, PHP 5.3.1, LibXSLT 1.1.24 (PHP memory_limit global/local 128M/128M)

The first one dies with more than 1032 entries, the second one with slightly fewer (!) entries.

[EDIT]: Both test systems were not using the extacly same datasource. It was a custom-made DS on the first machine, and a static DS on the secong one. This might explain a slight difference.)

According to the changelog, there were no fixes to the xsltprocessor after 5.3.0.

I am wondering if no other Symphony user has experienced this. My XML is around 6000 lines, containing around 1500 very simple items like this one:

    <entry>
        <e-mail>email@example.com</e-mail>
        <uid>1148a7fbe51bd1fa</uid>
    </entry>

All I do in XSLT is an xsl:copy-of on the parent element.

[EDIT]: It’s Apache 2.0 in both cases, by the way.

Do you think it could be this bug that was fixed in PHP 5.3.2…

Fixed bug #49660 (libxml 2.7.3+ limits text nodes to 10MB). 

Hmmm, nope. My text nodes are much smaller than this!

Maybe this one:

Fixed bug #50519 (segfault in garbage collection when using seterrorhandler and DomDocument).

I’ve seen that Symphony does this in class.xsltprocess.php.

Nope. I commented out all the set_error_handler and corresponding restore_error_handler lines — no change.

F…ck! It works on the Mac OS X machine using default installations of Symphony 2.0.7 (slightly patched, JavaScript issue…) and 2.0.8 RC1.

Now all I have to do is find the difference. :-(

I’ll search around and see if I can find some bugs on Mac versions of Apache.

Forget this! Using default installations I just pushed it to over 45.000 lines of XML without any problems! So now I have to think what might be the problem with those “real” websites…!

Here is what I have so far:

Test installations:

  • a default workspace installation of Symphony 2.0.7 (website #1)
  • a default workspace installation of Symphony 2.0.8RC1 (website #2)
  • a complex website using Symphony 2.0.7, featuring lots of pages, DSs etc. (website #3)
  • another complex website using Symphony 2.0.7, featuring lots of pages, DSs etc. (website #4)

The first three installations are on local development server (OS X, Apache 2, PHP 5.3.1). The last one is on a live server (Debian, Apache 2, PHP 5.2.0-8).

In every one of these three test environments I created a static DS and copied my XML there. I created a page, attached this single DS and nothing else. With a simple xsl:copy-of in the page’s XSLT I tried to parse the entries to XML. (This may seem rather useless, of course — but testing requires simplification…)

What happens is:

  • website #1 is definitely able to parse more than 100.000 lines of XML (25.000 entries)
  • website #2: like website #1
  • website #3: the page breaks (as described above) for a little more than 4000 lines of XML (1000 entries)
  • website #4: like website #3

I could not test more than 100.000 lines of static XML because saving a bigger static DS did not work.

So I think there is a general problem (memory leak?) in Symphony. On websites with many pages, DSs, events (or whatever) Symphony isn’t able to parse lots of XML data.

Alistair? Help!

I found the problem!

The name is: Datestamp Helper extension. If I uninstall this, everything is fine.

No wonder: it uses the FrontendOutputPostGenerate delegate to parse the complete output of the page using the preg_replace_callback() function. Seems like this takes too much resources.

That’s a pity: I love this extension…

I was searching to see if I could help you find a solution. I’m glad you found it.

Curious, the 100,000 line xml, is that a static file you are bringing in? If so, did you consider installing Rowan and Nick’s XML importer? You might have to import it in chunks though since the extension doesn’t seem to handle really large xml imports (or that’s been my experience). But I can told you this extension is a gem. After setting it up to bring it in chunks, it literally, became a very easy process for me to import over 18,000 lines of xml, which was about 2800 records.

Still, not as many as you were talking about, but it might be an option for you. Just a thought.

Curious, the 100,000 line xml, is that a static file you are bringing in?

Nope. This was just for testing. :-)

As I mentioned in the first post, I discovered the issue when I tried to use an extension with a custom datasource (grabbing data from a different database). The whole thing should definitely work for some 10.000 entries! I decided that testing on my local machine would be easier with a static datasource (because the custom DS will work on the live server only).

Thanks for your help, Brian!

Ah, gotcha! Well, I’m glad that you found root of the issue.

Just a short note: So it turned out that XSLT transformation did not break — it was page output which was broken by the extension’s attempt to do a search and replace on thousands of lines of code.

Create an account or sign in to comment.

Symphony • Open Source XSLT CMS

Server Requirements

  • PHP 5.3-5.6 or 7.0-7.3
  • PHP's LibXML module, with the XSLT extension enabled (--with-xsl)
  • MySQL 5.5 or above
  • An Apache or Litespeed webserver
  • Apache's mod_rewrite module or equivalent

Compatible Hosts

Sign in

Login details