finding subdirectories from starting URL

Thread view:

Alan - 12 Nov 2007 00:20 GMT

I want to find subdirectories from a starting URL. For example,
if I start with http://www.someplace.net, I want to be able to find
the subdirectories there, e.g.:

http://www.someplace.net/documentation/
http://www.someplace.net/about/
http://www.someplace.net/images/

Are there Java methods that facilitate this?

Thanks, Alan

Reply to this Message

Daniel Pitts - 12 Nov 2007 00:32 GMT

> I want to find subdirectories from a starting URL. For example,
> if I start with http://www.someplace.net, I want to be able to find
[quoted text clipped - 7 lines]
>
> Thanks, Alan

There is no easy way to do that, unless someplace.net gives you a
listing page. Generally, in order to do that, you have to have either
direct access to the disk, access to an FTP account on that machine, or
you have to crawl the web site and parse out the urls.

Note, this is not a limitation of Java, but simply a result of the way
http works.

There are plenty of web-crawling libraries/programs out there, I suggest
you Google for them.

Good luck,
Daniel

Signature

Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

Reply to this Message

Sherman Pendley - 12 Nov 2007 04:40 GMT

>> I want to find subdirectories from a starting URL. For example,
>> if I start with http://www.someplace.net, I want to be able to find
[quoted text clipped - 15 lines]
> Note, this is not a limitation of Java, but simply a result of the way
> http works.

In particular, note that documentation, about, and images may not even be
directories at all. A content-management system could use them as keys
into a database of managed documents, for example, or as category keywords
that are used to dynamically assemble a list of documents in the specified
category.

sherm--

Signature

WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net

Reply to this Message

Andrew Thompson - 12 Nov 2007 03:11 GMT

>I want to find subdirectories from a starting URL. For example,
>if I start with http://www.someplace.net, I want to be able to find
>the subdirectories there, ..

Why? What business is that of yours?
What does this ability offer to the end user?

Signature

Andrew Thompson
http://www.athompson.info/andrew/

Reply to this Message

Roedy Green - 12 Nov 2007 03:15 GMT

>http://www.someplace.net/documentation/
>http://www.someplace.net/about/
>http://www.someplace.net/images/
>
> Are there Java methods that facilitate this?

In general no. It is considered confidential information. Sometimes a
server will give you a directory listing in HTML if you give it an URL
of a directory without in index.html file in it.

Signature

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

Reply to this Message

Lew - 12 Nov 2007 04:45 GMT

>> http://www.someplace.net/documentation/
>> http://www.someplace.net/about/
[quoted text clipped - 5 lines]
> server will give you a directory listing in HTML if you give it an URL
> of a directory without in index.html file in it.

In addition, many directories on the server hard drive, while they are
subdirectories of the web document directory, will not be accessible to public
clients. A classic example is the WEB-INF/ directory tree in Java EE apps,
but Apache .htaccess can also restrict directories.

Signature

Lew

Reply to this Message

Andrew Thompson - 12 Nov 2007 06:11 GMT

>>http://www.someplace.net/documentation/

...

>> Are there Java methods that facilitate this?
>
>In general no. It is considered confidential information. Sometimes a
>server will give you a directory listing in HTML if you give it an URL
>of a directory without in index.html file in it.

An ironic side note is that I'd like my server to *allow*
automatic 'directory listing' for dirs with no index.html,
but cannot figure how to achieve it using the arcane
..contol panel ..thingy the host offers.

If directory indexing *is* turned on, it is a perhaps tedious
but mundane task to parse the resulting HTML, looking for
links to sub-dirs and resources.

Signature

Andrew Thompson
http://www.athompson.info/andrew/

Reply to this Message

Chris ( Val ) - 12 Nov 2007 06:33 GMT

> >>http://www.someplace.net/documentation/
> ..
[quoted text clipped - 8 lines]
> but cannot figure how to achieve it using the arcane
> .contol panel ..thingy the host offers.

[snip]

If it is an IIS web server, you will usually find a checkbox
in the IIS server properties, or for memory, you can even get
to it by right clicking on the virtual directory and editing
the properties.

Tomcat for example has the following setting in its web.xml file:

<init-param>
<param-name>listings</param-name>
<param-value>true</param-value>
</init-param>

But I think it's only good whilst developing.

--
Chris

Reply to this Message

Alan - 12 Nov 2007 14:37 GMT

Thanks for the information. I think I shall just follow href
links instead of finding directories.

Thanks, Alan

Reply to this Message

Andrew Thompson - 12 Nov 2007 16:02 GMT

>Thanks for the information. I think I shall just follow href
>links instead of finding directories.

In that case, as Daniel mentioned, search "Web crawler"/
"web crawling". There have been some interesting discussions
about crawlers in these groups, across the ages. As I vaguely
recall there was a source posted for one by Mr Omar Khan
..ahh yes.
<http://groups.google.com/group/comp.lang.java.programmer/msg/df4a6f43d57e3e6a

But please (please, please) respect the directions of the
site's robots.txt (if it has one).

Signature

Andrew Thompson
http://www.athompson.info/andrew/

Reply to this Message

finding subdirectories from starting URL

Free Magazines