Technical challenges streaming data over HTTP

I have recently been exploring options for enhancing the functionality of an existing website by integrating some streaming data feeds.

In the interests of making this functionality available on a wide range of devices (read iPads and iPhones as well as PCs) I would like this to be achieved without Flash or any other proprietary plug-in technology.

So, my old friend and foe Javascript is the lead candidate for providing the functionality on the client side.

Almost every article and blog post that I have read on the subject emphasises that client side polling is not appropriate, and that instead a continuous connection must be kept open between the web browser and the http server.

To avoid some of the challenges posed by Javascript's cross site scripting (XSS) security restrictions we would ideally push the data feeds out from the same domain name as the page that contains the feeds. For less data intense AJAX functionality this was made possible by hosting the servlet engine behind apache httpd.

Historically the way that most http servers have served their content involved dedicating a thread or process to serve each incoming request from start to finish. This does not scale well when incoming requests need to hold a connection over a prolonged period of time, so alternative approaches such as Comet have been implemented.

The Tomcat implementation of Comet is not supported when Tomcat is hosted behind apache httpd over AJP.

I was considering keeping the data serving system physically separate from the serving of HTML content anyway, so this restriction is just giving a friendly push to try using a separate sub-domain.