Symphony.

We have reason to believe that Dynamic XML DS caching is not working as expected:

http://www.getsymphony.com/community/discussions/30583/ http://github.com/symphony/symphony-2/issues/#issue/90

Additionally, I am getting white pages on http://olli.phoque.de/fotografie/flickr/ every couple of requests (don’t know what it is, sounds like a caching-problem to me).

Can somebody take a look or explain how Symphony caching works?

I have done a series of tests locally and the Dynamic XML DS seems to work as intended. Not sure if it will help, but I’ve committed some tweaks here and here) that may or may not help.

Additionally, I am getting white pages on http://olli.phoque.de/fotografie/flickr/ every couple of requests

This may point to a lower level, server related, problem. But hard to know.

Perhaps you can do some testing on your server. Use this script - http://pastie.org/661442 and see if the DS XML is what you expect. The script is just as simple proxy for some other site. In this instance I used Twitter as the test case.

The curl-script runs just fine (takes a long time every once a while).

After deleting all lines except the _session_config in sym_cache, the lines were not re-created on reloading the affected page.

So, the XML is not getting cached?

Thats what I was implicitely trying to ask: does that (the missing lines in the table) mean its really not being cached?

ha. well, yeah, I guess it does mean the caching isn’t working. Question is, why?

If you clear out the cache table, then refresh the page in Symphony that is using a dynamic DS, what appears in the page XML? and what about the table? then, try refreshing the page again. See if the creation timestamp changes.

I cleared out my cache table but it didn’t do anything, but I have noticed some other odd behavior if it helps.

I am pulling various feeds into my site to test, and on every refresh the status is set to “fresh” with a newer timestamp. The data itself doesn’t appear to be updated though. If I pull in my Delicious feed and then add a new link it will show up on their feed, but not in the symphony debug, even though the status is fresh and shows the date and time of the page refresh (unless I’m misunderstanding what those mean).

That being said, I’m pretty sure a request is actually being send to the server as I hit the Twitter rate limit almost every hour, and that has never happened until I tossed the Twitter feed onto my symphony page, if someone not on shared hosting can confirm this that would help too.

I’ll see if I can test against an rss feed from a site I own and check if there is actually a request being sent.

Hm… the XML returned is <flickr-photoset-list><error>No records found.</error></flickr-photoset-list>.

I am trying to load flickr.photosets.getList from the Flickr API: http://api.flickr.com/services/rest/?method=flickr.photosets.getList&api_key=697a277044d00ec2c28162069a8ae2fa&user_id=7328929%40N07&api_sig=36eda4847b8fbd1dae38f6666b8b0e2d

That could mean your xpath expression is not returning any nodes. If that happens, there is nothing to cache, so it will try again every reload. If you set your xpath expression to something simple, like /, do you get anything?

Sorry, I forgot to mention that the Xpath expression is /.

okay, well, No records found. indicates that there is no XML getting returned from the curl request. I tried the url you posted above, and everything is working fine. Are you testing this locally, or on a live server somewhere?

Public, on http://olli.phoque.de/

At frist, I was able to import Flickr RSS Feeds without problems, then I got white screens every now and then. Now I can’t import those REST methods at all.

Short of debugging your site myself, not sure what else to say. Randomly dropped connections makes me think there is a problem with your server’s installation of CURL. Perhaps it is seg faulting. Do you want me to take a closer look at your install?

One last thing, does the site work locally okay?

I have just exported the ensemble and installed it locally (love that feature btw!). Its working perfectly fine.

I’ll send you a PM with access data to my server.

On my local machine, a regular DS execution takes between 0.005 and 0.02s, dynamic XML DS however, take about 0.4 to 0.5s.

sym_cache entries are recreated on reload, this means caching seems to be working.

Doesn’t make sense to me…

Just to clarify, you’re saying the 0.4 - 0.5s load happens all the time?

Yes, on every request.

I have extracted a few DS line and the Gateway:

• http://olli.phoque.de/curltest.php
• http://olli.phoque.de/gateway.php

Sources:

• http://olli.phoque.de/curltest.phps
• http://olli.phoque.de/gateway.phps

Regular curl works just fine:

• http://olli.phoque.de/test.php
• http://olli.phoque.de/test.phps

Found the problem. Line 55 of class.gateway.php parses the URL. Looks like:

$this->_host = $url_parsed['host'];

This is then used later on, on line 127:

curl_setopt($ch, CURLOPT_URL, $this->_host . ':' . $this->_port . $this->_path);

However, this produces URLs like api.flickr.com/blah.... Notice the missing http://. Ive noticed this before, but on some servers (potentially those with an outdated version of CURL), a missing scheme causes weird errors. Specifically blank pages, indicating a seg fault.

The simple way to fix it is to change line 55 to be:

$this->_host = $url_parsed['scheme'] . "://" . $url_parsed['host'];

But I would be curious to know if just updating CURL to the latest version would do the trick.

Oh, I should mention, whatever the case I will update the gateway class to be more robust.

Thanks, it’s working now!

Still 0.5 seconds for each dynamic XML DS though.

Right. well, at least curl is not throwing blank pages now. I’ll see if I can figure out why the caching is not working as expected.

Looks like the problem was a commented out line in class.cacheable.php

        //if(!$data = @base64_encode(@gzcompress($data))) return false;

This a problem localised in phoque’s installation. Does not appear in the master, integration or downloadable copy of Symphony.

Going to close this issue. I’ve opened a new one for the missing scheme in curl requests: http://github.com/symphony/symphony-2/issues/#issue/156

I know this issue is closed, but found it because I had the same problems. I found two more things: 1) The 6 second request timeout was being hit when trying to get twitter RSS. Maybe the timeout is making some requests fail? Maybe the timeout needs to be increased?

2) I had to remove the forced port 80 in addition to adding the URL scheme–I have an RSS feed delivered over HTTPS (don’t ask why… long story!), and port 80 obviously doesn’t work with HTTPS (should be 443).

Not sure if those fixes should also be made?

After thinking about it… adding the scheme doesn’t fix the caching issues but only the not-loading-issues with older cURL versions, right?

Yeah, it didn’t fix my caching issue, but did fix the HTTPS issue I had. I figure the scheme should be in there for correctness, though…

I know this issue is closed, but found it because I had the same problems

You had the same problems as phoque? I.E. cache never getting created and blank pages? or you just had the 2 problems you mentioned below?

1) The 6 second request timeout was being hit when trying to get twitter RSS. Maybe the timeout is making some requests fail? Maybe the timeout needs to be increased?

It should really be a configurable value. Either the DS editor needs to have field for setting timeout, or there needs to be a config setting in /manifest/config.php

2) I had to remove the forced port 80 in addition to adding the URL scheme–I have an RSS feed delivered over HTTPS (don’t ask why… long story!), and port 80 obviously doesn’t work with HTTPS (should be 443).

You’re right. I think the port should only be explicitly set when it is present in the supplied URL. E.G. http://blah.com/script.php would not have the port set, letting CURL default to a port based on scheme, where as http://blah.com:1234/script.php would have it set to 1234.

adding the scheme doesn’t fix the caching issues but only the not-loading-issues with older cURL versions, right?

It works 2 fold. If you do not set the port, the scheme is used to figure out the port. E.G. FTP would be 22, HTTPS 443 etc.

Not never caching, but periodically returning a blank document, yes. And usually only with Twitter, as somebody mentioned on one post I read.

Maybe the Dynamic XML datasource should return the last fetched good document if the server responds with bad/no XML?

(Agree on all your other points about ports, etc.)

That would be a great feature to have, if it comes back with a feed that perhaps doesn’t validate, it wouldn’t update the datasource, that would solve a few of my problems.

I don’t mean this as a leading question, because I really don’t know what the answer is, but maybe some of this has already been figured out by the SimplePie project? http://simplepie.org/

Maybe the Dynamic XML datasource should return the last fetched good document if the server responds with bad/no XML?

Actually, that’s the current behaviour. Until recently though, the DS was not testing the status code returned. Only that the string returned was valid XML. If you take a look at this integration branch code you’ll notice that it now also tests for a 200 OK in addition to checking for valid XML. Should the status code be something other than 200 OK or the XML is invalid, than the cache will be used (if it exists) and failing that an <error> element placed in the output.

Hmm… I definitely get an <error> element in the output. And the feed has definitely been loaded before by the site, so something’s not right then… unfortunately, I can’t always reproduce it. :( Maybe the Mutex can’t be acquired? I noticed there’s some thread stuff going on in there?

I get an error element in the output as well. Twitter also returns 400 if you hit the rate limit.

Are you using 2.0.6 or integration branch code?

Sorry, I am using 2.0.6 I just read right over the fact that it’s in a different branch.

Well, the problems you mention should have been fixed. If you get a chance to test them out, that would be great.

I have also created 2 new issues #157 and #158 to address some concerns raised by jeffmaki.