Searching on Googlewith Java

Thread view:

Marcelo - 07 Nov 2005 15:25 GMT

Dear Programmers,

I was trying to extract images automaticly from Google with an URL request.

My problem is that, whenever I give to Google the URL of my query,
something like

String url = "http://images.google.ch/images?q=auto&hl=en";

there is no connection between my Java Program and the requested page.

I suppose that Google have a sort of protection, but actually I am not sure.

What do you think?

Thanks

Marcelo

PS: If I do the some thing to Yahoo, there is no problem.

Reply to this Message

Andrew Thompson - 07 Nov 2005 15:45 GMT

> What do you think?

<zen answer>
<http://groups.google.com/group/comp.lang.java.programmer/search?q=google.com+url
connection&start=0&scoring=d&>
Fight fire with fire.*
</zen answer>

* Somewhere in the threads of the top 3-4 hits in that search,
should be the answer to your question.

HTH

Reply to this Message

Tris Orendorff - 07 Nov 2005 17:57 GMT

> Dear Programmers,
>
[quoted text clipped - 7 lines]
>
> there is no connection between my Java Program and the requested page.

Try the Java API for Google at: www.google.com/apis/download.html

Signature

Sincerely,

Tris Orendorff
[Two antennae meet on a roof, fall in love and get married. The ceremony
wasn't much, but the reception was excellent.]

Reply to this Message

Roedy Green - 08 Nov 2005 04:01 GMT

On Mon, 07 Nov 2005 17:57:06 GMT, Tris Orendorff
<triso@remove-me.cogeco.ca> wrote, quoted or indirectly quoted someone
who said :

>Try the Java API for Google at: www.google.com/apis/download.html

this has a rather onerous licensing restriction:

PERSONAL AND LEGITIMATE USES ONLY
The Google Web APIs service is made available to you for
your personal, non-commercial use only (at home or at
work). You may only create a single account and must
provide accurate identification, contact, and other
information required as part of the registration process.
You may not create any script or other automated tool
that attempts to create multiple Google Web APIs accounts.
And you may not use the search results provided by the
Google Web APIs service with an existing product or
service that competes with products or services offered
by Google.

It does not make sense for Google to offer an API since they make
their money displaying ads. Perhaps this restriction is a temporary
one while the API is in beta.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Reply to this Message

Andrew Thompson - 08 Nov 2005 04:17 GMT

> On Mon, 07 Nov 2005 17:57:06 GMT, Tris Orendorff
> <triso@remove-me.cogeco.ca> wrote, quoted or indirectly quoted someone
[quoted text clipped - 3 lines]
>
> this has a rather onerous licensing restriction:

I do not think that is particularly onerous, given
what Google is offering for free.
...

> It does not make sense for Google to offer an API since they make
> their money displaying ads.

(shrugs) If the availability of the API encourages development
of commercial uses of Google (I understand Google limits the
number of searches you can do per day for free, but you can
pay for extra searches).

>..Perhaps this restriction is a temporary
> one while the API is in beta.

I do not see any reason why Google whould choose to
change that. They are a commercial entity, not a charity.

[ Oh, and yes, yes.. I do realise the information that
Google indexes belongs to the page authors rather than Google,
that is irrelevant. We could (conceivably) make our own
web crawler/search bot/indexing engine and end up with the
results we want, but if we choose to use Google's index, it
is obviously the index itself that is the 'added value' here. ]

Reply to this Message

Roedy Green - 08 Nov 2005 04:40 GMT

On Tue, 08 Nov 2005 04:17:01 GMT, Andrew Thompson
<seemysites@www.invalid> wrote, quoted or indirectly quoted someone
who said :

>I do not think that is particularly onerous, given
>what Google is offering for free.

This is little point in writing code that can be used by only one
person, and/or that cannot be sold.

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.

Reply to this Message

Andrew Thompson - 08 Nov 2005 05:19 GMT

> On Tue, 08 Nov 2005 04:17:01 GMT, Andrew Thompson
> <seemysites@www.invalid> wrote, quoted or indirectly quoted someone
[quoted text clipped - 5 lines]
> This is little point in writing code that can be used by only one
> person..

As I understand it '1 account' does not mean '1 person'.

If I expose the Google API via my web pages, I
('WebMaster') have '1 account' with Google, but
my 'n' users do not require any form of Google account,
just the link to my web page. My users can then make a
(very limited) number of searches before Google
says 'no more for today - unless WebMaster upgrades to..*'

* ..and that is where the commercial aspect comes in.

(shrugs) As I understand it (based mostly on the perhaps
flawed conclusions above), it can work quite well for
a specialised search of your own site, and if you hit
the limit of daily searches on a regular basis, it might
be worth considering the upgrade.

Personally, I am happy to use the kind of search I suspect
you have on your site. A little search box that specifies
our own domain name in a hidden field, and dumps the visitor
to the Google search page, complete with the Google adds (I
assume my users know roughly what they are looking for, and
hope they are not too easily distraced by the Google ads!)

Apparently you can even 'inform' the Google results
page to adopt the styles of your own site, but I have
never bothered taking it that far.

> .., and/or that cannot be sold.

Again - I believe you *can* commercialise your Google
related products, but Google (understandably) wants some
form of 'cut' of that, given it is their index which feeds
current data to the application.

Of course, IANAL, that is just my current understanding.

Reply to this Message

sks - 07 Nov 2005 18:23 GMT

> Dear Programmers,
>
[quoted text clipped - 18 lines]
>
> PS: If I do the some thing to Yahoo, there is no problem.

Google blocks connections with the user agent header set to Java, so you
need to override that. The HttpURLConn adds its automatically.

Reply to this Message

Marcelo - 07 Nov 2005 19:22 GMT

Thank you very much,

the URLConnection was a very elegant solution

Marcelo

Reply to this Message

Searching on Googlewith Java

Free Magazines