Symphony.

I thought cron jobs happened at set intervals? So either you’d have to hide a post until after the cron job had run or you’d have to post it with the data not fully live as the files might not be fully available.

Correct. Hence my comment above that s3cmd is great when the upload is asynchronous to the files needing to be on S3. Writing an extension of the File Upload field would allow this to be synchronous — as soon as you have uploaded to the server it gets pushed up to S3 without waiting for a schedule to catch it.

The potential issue with this is that the natural delay that exists in getting files to S3 will still result in broken files, audio and video on site:

• Connect time
• Upload time
• Directory read time
• File comparison time for large directories
• S3 setup time
• S3>Cloudfront push delay

Certainly on sites with lots of video, the update process can be very long - often in the 10’s of minute’s. What would be ideal is an extension that serves files from the local server until they are available on Cloudfront. This would require some form of call back from s3cmd to the extension, but that could be built in easily enough.

Has anyone successfully built an extension for Amazon S3?

Nope, I never got around to it. I have toyed with the idea of an Ankoder Video Field that pushes video files to Ankoder for encoding and hosting in the cloud, but I’ve never had the need for it.

On a side note, its a known practice that using a content delivery network (a collection of web servers distributed across multiple locations to deliver content more efficiently to users) speeds up load times of static files.
Then would it be usefull to host all your pictures on google (picasaweb) for the advantage of ;

1. different domain (as mentioned in this thread)
2. aren’t the google servers setup as CDN?…..

I would think when I browse my google photogallery in say, France, that it comes from a google server in France, and not from USA…

ofcourse the TOS of google might be problematic…

Picasa might be problematic but Flickr wouldn’t. Flicker specifically allows you to hotlink to your stuff.

Better yet, it allows you to grab your image feed as XML.

@dougoftheabaci
Picasaweb has a great api and imagefeeds, to use and ‘hotlink’ your pics.

But either flickr or picasaweb, the question remains:
Is using them for imagehosting a way of getting a free CDN, and as such does it makes sense to use them for ‘serious’ websites.

I would say it depends on the intended use. If use use them as parts of your actual website, such as the background images, then that’s something I wouldn’t suggest.

However, if you happen to have a photoblog or want to have photos of events and pull them in to your website from an external service that’s perfectly fine, in fact that’s recommended. It’s the same as someone posting a video blog on YouTube but pulling those videos in to their own site.

It’s not only good for site speed but it’s good for SEO and for users as it gives them more ways to interact with you.

But either flickr or picasaweb, the question remains:
Is using them for imagehosting a way of getting a free CDN, and as such does it makes sense to > use them for ‘serious’ websites.

AWS (amazon) offers a CDN as well (for S3). You might consider this if you need a certain responsiveness. I’d rather not rely on these “web 2.0” websites (running on 100% load most of the time, sometimes more than this).

You could also potentially use Google’s AppEngine as a CDN.

@makenosound
Great tip, so I include that beyond appengine all other google webapps also aare CDN based, despite the fact I can’t find this proven anywhere.

@dougoftheabaci

If use use them as parts of your actual website, such as the background images, then that’s something I wouldn’t suggest.

Could you clarify why exactly?
Just because layout images dont have any meaning to users, or because in the end they still load faster from your own server, even if its accross the ocean, or because my server has 99.9 uptime, and google doesn’t?

Has anyone used a CDN (S3, google appengine,…) with a symphony project, and any findings to share about it?

I am looking for data about how responsive google apps really is or isn’t compared to a shared hostingaccount accessed from accross the ocean.

S3 has buckets in Europe and US, what about google apps? Also worldwide or just US?

S3 has buckets in Europe and US

More precise: The AWS CloudFront for S3 has edge locations in USA, Europe, Hongkong/Singapur and Japan.

OK so I should test the pageloading times from different visitors worldwide.
I guess ab split and GA timetracker would be the tools to use… anyone seen interesting cases/tutorials for such multi country testing?

Could you clarify why exactly?

FIrstly, because you’re relying on a third-party service for something that your site relies on. If the pictures go down for an hour or two the site is still there and that’s what’s important.

Secondly, unlike the images, site assets are accessed a number of times and ar cached the first time so you won’t get as much of the benefit the way you would with content images.

Thirdly, as you point out, the images won’t stand alone like photo content would. A picture of a group of people, when removed from your site, is still a group of people.

I like the idea of uploading the images used in posts and galleries to Flickr because of the fact that it has dual-use. Not only does it provide a CDN for your site but it has the further benefit of giving users a way to download, link to and comment on those photos. A background sprite doesn’t have the same benefit.

thanks dougoftheabaci,
inspired by your second point, I cannot find the thread anymore, but recently read that the JIT extension doesn’t allow browser caching. For site speed optimisation that sounds not perfect, is there a way to modify it?

Here ist the link to the JIT and browser caching post. No solutions were found there. You could use some other on-the-fly image editing script that supports browser caching.

If the images aren’t cached anyway then that would be more of a reason to use Flickr.

Not only do you provide more options for your users but you speed up your site’s load time by offsetting some of the strain to Flickr’s servers.

I’m going to start looking into this for further use. I might just have to use this more myself.

Picasweb’s api offers JIT too, but its a bit limited, so sometimes I use symphonies JIT, but I would be better of using css to further crop the images from google.

Also I think our should look into adding a database with fallback images, because our JIT images expire (would be better to never expire),or if no JIT is used, if anything goes wrong with the external datasource… Maybe I can use the dummy extension, but this too is an external service…

I like the idea of uploading the images used in posts and galleries to Flickr because of the fact that it has dual-use. Not only does it provide a CDN for your site but it has the further benefit of giving users a way to download, link to and comment on those photos. A background sprite doesn’t have the same benefit.

True. I’m going to nitpick and say this technically isn’t a CDN, but is simply sharing the load between your site and another.

There are several reasons for why a true CDN is beneficial:

• you serve content from a different (sub)domain. Browsers limit the number of concurrent requests they make to each domain, therefore serving assets from another domain (or subdomain) means your browser can make more simultaneous requests and assets are downloaded faster (in parallel, not serial)
• assets served via a CDN are more likely to have good cache/expiry headers set (the CDN provider will configure these) such as ETags, so that assets are cached by the browser for longer thereby improving performance
• assets served via a CDN are generally done so from a distributed network of servers (hence the name). Amazon S3 (buckets) can be served through an additional layer called Cloudfront, which essentially means your files from S3 are cached on servers across the globe and served from the location that is most appropriate for the user. This serves to reduce latency and hopefully a faster download speed for that asset

I use Amazon S3 and Cloudfront on Bekonscot where I manually upload my CSS, JS and Flash assets to an S3 bucket. To make this transparent, I have .htaccess rules such as:

RewriteRule ^assets/files/(.*)$ http://media.bekonscot.co.uk/files/$1 [L]

Which catch any requests to my /assets/files directory and map them onto media.bekonscot.co.uk. This in turn is a CNAME onto my Cloudfront account. Serving the files in this way gives me the three benefits listed above: more simultaneous downloads by the browser, correct ETags (expires/cache headers) and geographically distributed delivery.

Of course, this only works currently with my static assets. But there is an active thread on this forum about developing a File Upload field with the capability to push the asset to Amazon S3 itself.

There might be a simpler way of achieving S3 integration by using an .htaccess rule to catch the JIT requests and map these onto S3/Cloudfront directly. I’m going to have a think about that!

@Nick, now I am confused…

I believe google does serve his images from different server farms accross the globe, hence I would call this true CDN. Can you/anyone confirm this? Ea the paths to images always are formed ; http://lh6.ggpht.com with lh6 ranging from one 2 6…

And even if all google images come from one server, its still another one as your own (point 1)

Point 2; expiry headers, doesnt google picasaweb does this too?

Point 3, again isnt this the case with google?