When I try the code below with "http://www.google.com/search?
source=ig&hl=en&rlz=&q=something" in the URLs.txt file, I get an
IOException error. The server is returning an HTTP response code
403. However, when I open the same URL in my browser, it works
fine.
A different but similarly formated URL to another server worked
fine. Maybe this server is expection something additional?
I am new at this, and I wonder if it is obvious what the
problems is.
Thanks, Alan
// download text content of URL
import java.net.*;
import java.io.*;
public class URLget
{
public static void main ( String[] args ) throws IOException
{
try
{
BufferedReader infile = new BufferedReader(new
FileReader("URLs.txt"));
PrintWriter out = new PrintWriter(new
FileOutputStream("downloaded.html"));
String aURL, str;
while ((aURL = infile.readLine()) != null)
{
URL url = new URL(aURL);
System.out.println(aURL);
BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream()));
while ((str = in.readLine()) != null)
{
out.println(str);
}
in.close();
}
infile.close();
out.close();
}
catch (MalformedURLException e) {e.printStackTrace();}
catch (IOException e) {e.printStackTrace();}
}
}
Arne Vajhøj - 22 Oct 2007 04:07 GMT
> When I try the code below with "http://www.google.com/search?
> source=ig&hl=en&rlz=&q=something" in the URLs.txt file, I get an
[quoted text clipped - 4 lines]
> A different but similarly formated URL to another server worked
> fine. Maybe this server is expection something additional?
I think Google tests on browser type.
In 2004 the following worked:
URL url = new URL("http://www.google.dk/search?q=hej");
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible;
MSIE 5.01; Windows NT 5.0)");
con.setRequestProperty("Referrer", "http://www.google.dk/");
con.connect();
if(con.getResponseCode() == HttpURLConnection.HTTP_OK) {
InputStream is = con.getInputStream();
byte[] b = new byte[1000];
int n;
while((n = is.read(b)) >= 0) {
System.out.println(new String(b,0,n));
}
is.close();
} else {
System.out.println(con.getResponseCode() + " " +
con.getResponseMessage());
}
con.disconnect();
Arne