RSS and Polling, again

Don’t forget that there are drawbacks to the common methods of reducing RSS bandwidth.

Method #1 is to reduce the polling frequency – but if CNN tells my aggregator to poll every 4 hours (because they don’t like paying for 50 million hits on their server every 15 minutes or even every hour), there’s a good chance I’m going to visit the site myself and see stories that I haven’t seen in my aggregator yet. Once users see that, their confidence in RSS drops.

Method #2 is to put less text in the RSS items (I understand this is what MSDN has done). But that just makes the RSS feed less useful. I get a headline and I have to hit the site to see if I’m really interested in the content or not.

There are applications of RSS that would benefit from real-time publishing, and certainly everyone would like to see new items as soon as they’re posted (even if they don’t read them right away). With a Jabber style protocol the publisher uses zero bandwidth until they publish something, and then they send a small XML snippet to a Jabber server which, in conjunction with a network of other Jabber servers, informs all the interested users that there’s a new item.

The problem here is that now 50 million users try to hit the URL at the same time and kill it :) That’s why I prefer an NNTP style approach where my aggregator polls a relatively local server with a higher frequency and has direct access to the RSS item without having to hit the source site for it.