 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / November 20052 question poll: large and long http response (stream like)
Hi, We are arguing about the elegance/irregularity of the possibility to send *gigs and gigs of data *during many days *over a single https response. *[with tomcat5 & java 1.4.2] Assuming the client can handle it and that the tcp/ssl layer are reliable and won't disconnect, would you say it is: question 1 a-against the http spec (what section______) b-supported/covered by http spec (chunked encoding) c-other________ question 2: a-risky (reason:_______) b-unusual (reason:_______) c-safe (reason:_______) d-common (example:_____) e-other_________ Thanks to all in advance. > Hi, > [quoted text clipped - 10 lines] > a-against the http spec (what section______) > b-supported/covered by http spec (chunked encoding) I'd say it would be supported using chunked encoding. > c-other________ > > question 2: > a-risky (reason:_______) What happens if there IS a disconnect for some reason? Can you resume operations where you left off? > b-unusual (reason:_______) Not unusual, per se. Internet routers, for example, create long-lived TCP connections to exchange routing information. The volume may be greater, but that's no matter. > c-safe (reason:_______) Dunno about safe. See question about disconnects. > d-common (example:_____) > e-other_________ > > Thanks to all in advance. >> We are arguing about the elegance/irregularity of the possibility to >> send *gigs and gigs of data [quoted text clipped - 10 lines] > > I'd say it would be supported using chunked encoding. I agree. The spec does not place any limit that I know of on the total size of a message delivered via the chunked transfer coding. It says little or nothing about the permissible time to deliver a complete response. >> c-other________ >> [quoted text clipped - 3 lines] > What happens if there IS a disconnect for some reason? Can you resume > operations where you left off? Spot on. A service interruption probably means that you're hosed, because as far as I can tell, there is no way to resume an HTTP transfer in the middle. You could build a resumable protocol on top of HTTP, but then you wouldn't be delivering the whole logical response in a single HTTP response. >> b-unusual (reason:_______) > > Not unusual, per se. Internet routers, for example, create long-lived > TCP connections to exchange routing information. The volume may be > greater, but that's no matter. I disagree. Routers may establish long-lived connections amongst themselves, but that's not the same thing as a continuous data stream over a comparable period. At least it doesn't seem that way to me. The proposed behavior is in any case assuredly unusual in the HTTP space.  SignatureJohn Bollinger jobollin@indiana.edu >>> We are arguing about the elegance/irregularity of the possibility to >>> send *gigs and gigs of data >>> *during many days >>> *over a single https response. >>> *[with tomcat5 & java 1.4.2] >>> question 2: >>> a-risky (reason:_______) [quoted text clipped - 7 lines] > then you wouldn't be delivering the whole logical response in a single > HTTP response. Well, HTTP DOES have support for "partial content", for example, see the "206 Partial Content" response status, and the "request ranges" support defined in RFC2616 (I forget the exact header ;-) >>> b-unusual (reason:_______) >> [quoted text clipped - 6 lines] > over a comparable period. At least it doesn't seem that way to me. The > proposed behavior is in any case assuredly unusual in the HTTP space. Essentially, I think that the OP is wanting to do something that requires lots of data to be transferred, and wants to do it without getting the firewall administrators involved. Hence the desire to do this on top of HTTP . . . Not necessarily a good idea, IMO. Rogan > Essentially, I think that the OP is wanting to do something that > requires lots of data to be transferred, and wants to do it without [quoted text clipped - 4 lines] > > Rogan Hehehe... As a coder, I would go straight to TCP. But yeah... I'm JACWABB (just a coder with a boss behind...) It's more like streaming data to simple clients, like applets, perl, wget to console, etc... Any easy-to-code data consumer. You're thinking: "but why don't you make many shorter connections?" Well, for now, we would really like not to 'skip a beat' because the streamed data in question is a flow of events. Having a 'mark' in the protocol (to resume at requested position) would suddenly mean extra bufferring and connection subscription/management code and ... aaaahhhhAAHH...(sigh...) while a non-expected loss of connection is not going to require lossless recovery. We would just tell the user, and he would restart. >>Essentially, I think that the OP is wanting to do something that >>requires lots of data to be transferred, and wants to do it without [quoted text clipped - 20 lines] > going to require lossless recovery. We would just tell the user, and he > would restart. One last thing to consider: If you expect to have some idle time during the transfer, and you will be going through proxies, you may also have to deal with the proxy thinking that the connection has gone idle, and disconnecting you. Many proxies will time out a persistent connection if there is no data trasferred for more than about 30 seconds . . . Rogan > Spot on. A service interruption probably means that you're hosed, > because as far as I can tell, there is no way to resume an HTTP transfer > in the middle. I thought there was some sort of request-range: header in HTTP 1.1 ? I admit that I'm too lazy to go look it up, so that may be complete bollocks... But even if it did exist, the question would arise: will the actual implementations work with ranges that can't be expressed in 32-bits ? -- chris >>Spot on. A service interruption probably means that you're hosed, >>because as far as I can tell, there is no way to resume an HTTP transfer >>in the middle. > > I thought there was some sort of request-range: header in HTTP 1.1 ? I admit > that I'm too lazy to go look it up, so that may be complete bollocks... Ah, um, yes; good point. Time to go back and reread the spec again. Request ranges might not really help, however, if the response is dynamically generated. That is especially true if it is a real-time stream. (E.g. data from a scientific instrument.) SignatureJohn Bollinger jobollin@indiana.edu > We are arguing about the elegance/irregularity of the possibility to send > *gigs and gigs of data > *during many days > *over a single https response. > *[with tomcat5 & java 1.4.2] I would worry about this. Specifically, I'd worry about 32-bit counters. At the TCP level there /shouldn't/ be a problem. TCP does have a 32-bit sequence number in it, but that is defined to wrap around. What /might/ be a worry is any other code that is in the transmission path, anti-virus software, anti-intrusion software, statefull firewalls, etc. The point is that although these things /should/ handle that much data, there's always a chance that they'll contain 32-bit counters that will cause problems if they wrap. I, personally, wouldn't worry /too/ much about this since I would expect the makers/writers of such tools to test long flows. I have no idea how a multi-gigabyte flow would interact with the SSL protocol. Nor what the security/crypto implications are (would you be effectively supplying an attacker with a very large body of data all encrypted with the same key ? I believe SSL has provision for negotiating a new key during a session, but I don't know when/how/whether that happens). At the HTTP level, there /shouldn't/ be a problem either (assuming chunked transmission). This time, though, I wouldn't feel nearly as confident that there would be no problems with 32-bit wraparound. Neither HTTP clients nor HTTP servers are designed /specifically/ for that kind of load, and I wouldn't be surprised if such conditions had received little or no testing (or, perhaps, even thought). The same considerations apply to any other s/w that is working at the HTTP level and which is involved in the execution and/or transmission path. -- chris >> We are arguing about the elegance/irregularity of the possibility to >> send *gigs and gigs of data [quoted text clipped - 32 lines] > > -- chris Thanks for the details. I keep my eye on tomcat: will the httpsession stay alive during that time, and if not, will the socket be forcibly closed... I think not, but eh, dunno. For sure, the CLF web log will look funny...! ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...! >. For sure, the CLF web log will look funny...! > ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...! And if it wraps to negative numbers, then your logfile analyser will have kittens ;-) -- chris > I keep my eye on tomcat: will the httpsession stay alive during that > time, and if not, will the socket be forcibly closed... I think not, but > eh, dunno. For sure, the CLF web log will look funny...! > ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...! This is a Tomcat implementation detail, and I don't have the familiarity with Tomcat's source that would be necessary to answer it. I find it highly unlikely that Tomcat would forcibly close the connection in the middle of a response, but it is possible that the user would end up in a new session if they had to reconnect after even a momentary interruption. This should be fairly easy to test. SignatureJohn Bollinger jobollin@indiana.edu Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.