Reading URLs with POST data vs. w/out POST

Thread view:

Hal Vaughan - 12 Jan 2008 03:49 GMT

I'm working a simple method to read web pages and experimenting with a few
aspects.

I've noticed if the URL contains POST data (and I'm just specifying the POST
data in the URL), when I try URLConnection().getContentLength(), I often
get a length of -1. I don't see this happen on any web pages without any
post data.

Is this because the page is generated dynamically and the server may not be
reporting the length for a posted page but is reporting it for a static
page?

I've tested different configurations in programs and even copied different
examples from web pages to test this out, but the effect is code
independent.

Here are two example pages:

Length reported correctly:
<http://www.archive.org/download/361003WorldSeriesGiantsVsYankees/361003WorldSeri
esGiantsVsYankees_files.xml>

Length reported as -1:
<http://www.archive.org/search.php?page=1&query=collection%3Aoldtimeradio&sort=title>

I don't think this is a Java language issue, but more a factor of what data
one gets back from a server. Am I right about this? Is it a server issue?
The server, according to Netcraft, is running Apache.

Hal

Reply to this Message

Arne Vajhøj - 12 Jan 2008 04:18 GMT

> I'm working a simple method to read web pages and experimenting with a few
> aspects.
[quoted text clipped - 23 lines]
> one gets back from a server. Am I right about this? Is it a server issue?
> The server, according to Netcraft, is running Apache.

When you talk about "URL contains POST data" I assume that you means
"URL with query string" (the data in a POST is not in the URL !).

Java docs for getContentLength() says:

#Returns:
# the content length of the resource that this connection's URL
# references, or -1 if the content length is not known.

The HTTP standard says about Content-Length header:

# In HTTP, it
# SHOULD be sent whenever the message's length can be determined prior
# to being transferred, unless...

It sounds very plausible that:
* the byte count can easily be detected for static content
* the byte count can not as easily be detected for scripts

Arne

Reply to this Message

Mark Space - 12 Jan 2008 22:02 GMT

> I'm working a simple method to read web pages and experimenting with a few
> aspects.

What happened to your last issue with setFocusableWindowState()? Did
you get it working?

Reply to this Message

Hal Vaughan - 14 Jan 2008 00:38 GMT

>> I'm working a simple method to read web pages and experimenting with a
>> few aspects.
>
> What happened to your last issue with setFocusableWindowState()? Did
> you get it working?

I had to set it aside for a bit and will be looking at that thread and that
issue in a day or so. That was more a cosmetic issue and I figured it was
more important to make sure everything was working properly before worrying
about UI issues.

Hal

Reply to this Message

Roedy Green - 13 Jan 2008 11:02 GMT

>Is this because the page is generated dynamically and the server may not be
>reporting the length for a posted page but is reporting it for a static
>page?

That is so, and often it is just left out presumably out of laziness.
See my code for
GET and POST at http://mindprod.com/products.html#HTTP

Signature

Roedy Green, Canadian Mind Products
The Java Glossary, http://mindprod.com

Reply to this Message

Reading URLs with POST data vs. w/out POST

Free Magazines