URLConnection "memory leak"

Thread view:

timjowers@gmail.com - 03 Jun 2005 05:51 GMT

URLConnection leaks memory and after a while will return the memory. No
amount of calling System.gc(); will clean it up. OK for small apps but
I am writing one that downloads 100 or so pages synchronously. OK
unless I experience several clients asking my server to do this at
once. Then Java gives the almost useless OutOfMemoryError. So much for
Garbage Collection. At this point GC just means random
non-deterministic memory management and means, unlike in C++, I cannot
achieve a robust server application. How can Java justify giving
OutOfMemoryError when NO REFERENCES ARE HELD IN MY PROGRAM!!!! I guess
URLConnection must have some internal static buggy code.

Any ideas how to get URLConnection to stop hanging onto memory?

Here's some test code below.

TimJowers

/**
* Sample memory
*/
import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;

public class Leaky {

    public String strXML = null;
    private String proxyHost=null, proxyPort=null, username=null,
password=null,
        httpRequest=null, base64Encoded=null, proxyType="2",
        formValues=null, formMethod="", optionFile=null;
    private URL url=null;
    private URLConnection urlConnection=null;
    private InputStream is=null;
    private File file=null;
    private FileOutputStream out=null;
    private int ftpPort=21;
    private boolean notAssigned = true,
        ftpTrue=false, ftpPassive=false, ftpAscii=false,
        https=false, nonverbose=true, post=false, booleanOptionFile=false;
    private StringBuffer sbuf = null;
    private byte[] bytesRead = new byte[4096];

    protected void zeroMem()
    {
        strXML = null;
        proxyHost=null; proxyPort=null; username=null; password=null;
        httpRequest=null; base64Encoded=null;
        formValues=null;
        formMethod=""; optionFile=null;
        url=null;
        urlConnection=null;
        is=null;
        file=null;
        out=null;
        sbuf = null;
        bytesRead = null;
    }

    public URLConnection setupProxy ( String [] args ) throws Exception
{
        getOptions(args);

        if (https) {
            if (!nonverbose) {
                System.out.println("Using https communication");
            }
            Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
            System.setProperty("http.keepAlive", "false");
            System.setProperty( "java.protocol.handler.pkgs",
                            "com.sun.net.ssl.internal.www.protocol" );
        }
        if (formMethod != null && formMethod.equalsIgnoreCase("get")) {
            httpRequest = httpRequest + "?" + encodeHttpString(formValues);
        }
        try {
            url = new URL(httpRequest);
        } catch (MalformedURLException e) {
            if (!nonverbose) {
                System.err.println("Malformed URL, must be
http://somehost[/file]");
            } else {
                throw new Exception("Malformed URL, must be
http://somehost[/file]");
            }
            return null;
        }
        if (proxyHost != null) {
            if (proxyType.equals("1") || proxyType.equals("3")) {
                System.getProperties().put( "proxySet", "true" );
                System.getProperties().put( "proxyHost", proxyHost);
                System.getProperties().put( "proxyPort", proxyPort);
            } else {
                if (proxyType.equals("4")) {
                    System.getProperties().put( "socksProxySet", "true");
                    System.getProperties().put( "socksProxyHost", proxyHost);
                    System.getProperties().put( "SocksProxyPort" , proxyPort);
                } else {
                    if (proxyType.equals("5")) {
                        System.getProperties().put( "ftpProxySet", "true" );
                        System.getProperties().put( "ftpProxyHost", proxyHost );
                        System.getProperties().put( "ftpProxyPort", proxyPort );
                    } else {
                        System.getProperties().put("firewallSet", "true");
                        System.getProperties().put("firewallHost", proxyHost);
                        System.getProperties().put("firewallPort", proxyPort);
                        System.getProperties().put("http.proxyHost", proxyHost);
                        System.getProperties().put("http.proxyPort", proxyPort);
                    }}}
            if (https && proxyHost != null) {
                System.getProperties().put("https.proxyHost", proxyHost);
                System.getProperties().put("https.proxyPort", proxyPort);
            }
        }
        try {
            urlConnection = url.openConnection();
        } catch (IOException e) {
            if (!nonverbose) {
                System.err.println("Error opening URL connection to " +
httpRequest);
            } else {
                throw new Exception("Error opening URL Connection to " +
httpRequest);
            }
            return null;
        }
        if (proxyHost != null) {
            Base64 base64 = new Base64(username + ":" + password);
            base64.encode();
            base64Encoded = "Basic " + base64.getOutgoing();
            urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
            if (proxyType.equals("3")) {
                urlConnection.setRequestProperty("Authorization",
                                                base64Encoded);
            } else {
                urlConnection.setRequestProperty("Proxy-Authorization",
                                                base64Encoded);
            }
        }
        return urlConnection;
}

    public boolean doWget ( String args ) throws Exception {
    // if args are all on one string then break into an array of strings
    StringTokenizer strTok = new StringTokenizer( args );
    String []argv = new String[ strTok.countTokens() ];
    for(int i=0; strTok.hasMoreTokens(); i++ )
        argv[i] = strTok.nextToken();
    return doWget( argv );
    }

    public boolean doWget ( String [] args ) throws Exception {

    System.out.println(httpRequest);
    System.out.println(proxyHost);
    System.out.println(ftpPort);

urlConnection = setupProxy ( args );

        if (!nonverbose) {
            System.out.println("Connecting to " + httpRequest + "...");
            if(
0==httpRequest.compareTo("http://www.p2p.wrox.com/listindex.asp") ) //
hangs
            return false;
            if(
0==httpRequest.compareTo("http://www.soton.ac.uk/~chst/direct.htm") )
// hangs
            return false;
            if(
0==httpRequest.compareTo("https://www.ahamembership.com/assoc6/k.cgi?p=10-TX&acct_code=TX004-10TX-T")
) // security exception
            return false;
            if(
0==httpRequest.compareTo("https://www25.hway.net/onl261/cgi-local/sgx/shop.cgi?page=order.html&afnum=21")
) // security exception
            return false;
            if( 0==httpRequest.compareTo("https://grc.com/x/ne.dll?bh0bkyd2") )
// security exception
            return false;
        }
        try
            {
            DataOutputStream    printout;
            DataInputStream    input;

        urlConnection.setDoInput (true);
        urlConnection.setUseCaches (false);
        urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
        urlConnection.setRequestProperty("Cache-Control","no-cache");

        if (null != strXML) // add XML data as HTTP payload
        { System.out.println("Adding XML...");
        // Send POST output.
        printout = new DataOutputStream (urlConnection.getOutputStream ());
        String content =
        "CoNUM=" + URLEncoder.encode ("2") +
        "&PASSWORD=" + URLEncoder.encode ("1");
            String msg;
msg = "<?xml version=" + "\"" + "1.0" + "\"" + "?>";
msg = msg + "</XML>";
    content = msg;
            System.out.println( " --- XML to Web Server ---\r\r" + content );
        printout.writeBytes (content);
        printout.flush ();
        printout.close ();
}
    }
    catch (MalformedURLException me)
        {
        System.err.println("MalformedURLException: " + me);
        }
    catch (IOException ioe)
        {
        System.err.println("IOException: " + ioe.getMessage());
        }
        if( !readHttp() )
        return false;
        closeAll();
        return true;
    }

    private void getOptions( String[] args ) throws Exception {
        for (int i=0; i<args.length; i++) {
            if (args[i].equals("-h")) {
                StringTokenizer st = new StringTokenizer(args[i+1], ":");
                int count=0;
                while (st.hasMoreTokens()) {
                    if (count == 0 ) {
                        proxyHost=st.nextToken();
                    }
                    if (count == 1) {
                        proxyPort=st.nextToken();
                    }
                    count++;
                }
                if (proxyPort == null) {
                    proxyPort = "80";
                }
                i++;
                notAssigned=false;
            }
            if (args[i].equals("-u")) {
                username=args[i+1];
                i++;
                notAssigned=false;
            }
            if (args[i].equals("-p")) {
                password=args[i+1];
                i++;
                notAssigned=false;
            }
            if (args[i].equals("-t")) {
                proxyType=args[i+1];
                i++;
                notAssigned=false;
            }
            if (args[i].equals("-post")) {
                formValues=args[i+1];
                i++;
                notAssigned=false;
            }
            if (args[i].equals("-method")) {
                formMethod=args[i+1];
                i++;
                notAssigned=false;
            }
            if( args[i].equals("-xml") ) // get XML file
            { strXML = new String("<XML></XML>");
            }
            if (proxyHost != null && ftpTrue==true) {
                if (!nonverbose) {
                    System.err.println("Proxy and Ftp cannot coexist use -h or -ftp");
                } else {
                    throw new Exception("Proxy and Ftp cannot coexist use -h or
-ftp");
                }
                return;
            }
            if (notAssigned) {
                httpRequest=args[i];
                if (httpRequest.substring(0, 5).equalsIgnoreCase("https")) {
                    https=true;
                }
            }
            notAssigned=true;
        }
    }

    /** Main invoked from the prompt.
    * @param args Arguments from the Stdin
    * @throws Exception Throws Exception
    */
    static String[] urls = {"http://www.youart.net/",
"http://www.microsoft.net/",
                            "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
                            "http://www.nbc.com/", "http://www.abc.com/",
                            "http://www.cbs.com/", "http://www.bbc.com/",
                            "http://www.ebay.com/", "http://www.yahoo.com/",
                            "http://www.ncr.com/", "http://www.ibm.com/",
                            "http://www.ford.com/", "http://www.cisco.com/",
                            "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
                            "http://www.cnn.com/"};
    public static void main ( String[] args ) throws Exception {

    for( int e=0; e<urls.length; e++)
    {    Leaky wget = new Leaky();
        wget.doWget( urls[e] );
        String result = wget.getOutput();
        System.out.println( "page has " + result.length() + " characters" );
        wget.zeroMem();
        wget=null;
System.gc();
System.gc();
System.gc();
    }
    // memory is leaked at this point. Next memory is magically returned
if we sleep awhile.
for( int s=0; s<60; s++ ) {
     try{ Thread.sleep(60000);}catch(Exception e){}
        System.gc();
}
}

Reply to this Message

Andrew Thompson - 03 Jun 2005 06:26 GMT

...

> public String strXML = null;

Please refrain form posting tab characters to usenet, as they are
often expanded to ridiculous lengths by news reader software. And
keep your line lengths short. Otherwise lines wrap, and therefore break..
...

> System.err.println("Malformed URL, must be
> http://somehost[/file]");

Also, you are missing a closing '}'. If you had stuck to a single
convention for formatting code, this might be easier to spot, but it
seems you are mixing and matching..

When those problems are fixed, I get..

C:\..\Leaky.java:60: cannot find symbol
symbol : method encodeHttpString(java.lang.String)
location: class Leaky
httpRequest = httpRequest + "?" +
encodeHttpString(formValues);
^
C:\..\Leaky.java:114: cannot find symbol
symbol : class Base64
location: class Leaky
Base64 base64 = new Base64(username + ":" +
password);
^
C:\..\Leaky.java:114: cannot find symbol
symbol : class Base64
location: class Leaky
Base64 base64 = new Base64(username + ":" +
password);
^
C:\..\Leaky.java:203: cannot find symbol
symbol : method readHttp()
location: class Leaky
if( !readHttp() )
^
C:\..\Leaky.java:205: cannot find symbol
symbol : method closeAll()
location: class Leaky
closeAll();
^
C:\..\Leaky.java:321: cannot find symbol
symbol : method getOutput()
location: class Leaky
String result = wget.getOutput();
^
Note: C:\..\Leaky.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
6 errors

...It seems you might benefit from a document specifically prepared
to help people prepare a good example of a problem. I suggest
(and recommend*) this one. <http://www.physci.org/codes/sscce.jsp>

* Not surprising, since I wrote and host it.

As far as the actual problem, I have no insights. I was curious
to see your code break, but do not have a great deal of experience
with URLConnection (at least, not in terms of memory leaks).

Signature

Andrew Thompson
http://www.PhySci.org/codes/ Web & IT Help
http://www.PhySci.org/ Open-source software suite
http://www.1point1C.org/ Science & Technology
http://www.LensEscapes.com/ Images that escape the mundane

Reply to this Message

Knute Johnson - 03 Jun 2005 16:01 GMT

> URLConnection leaks memory and after a while will return the memory. No
> amount of calling System.gc(); will clean it up. OK for small apps but
[quoted text clipped - 348 lines]
> }
> }

I have had problems in the past when I didn't close my
HttpURLConnection. You might try that.

Signature

Knute Johnson
email s/nospam/knute/

Reply to this Message

Alan Krueger - 04 Jun 2005 05:18 GMT

[snip 350-some quoted lines]

> I have had problems in the past when I didn't close my
> HttpURLConnection. You might try that.

Please try to trim quoted text before replying. Quoting hundreds of
lines of text isn't really necessary to give context for a 1-2 line
response.

Reply to this Message

Knute Johnson - 05 Jun 2005 01:00 GMT

> [snip 350-some quoted lines]
>
[quoted text clipped - 4 lines]
> lines of text isn't really necessary to give context for a 1-2 line
> response.

bite me!

Signature

Knute Johnson
email s/nospam/knute/

Reply to this Message

timjowers@gmail.com - 04 Jun 2005 20:00 GMT

Thanks for everyone's feedback. I determined
1) closing input stream speeds up memory recovery (closeAll below)
2) setting references to null speeds up memory recovery (zeroMem below)
3) calling gc speeds up memory recovery. Of course, at a cost but if
mem is the over-arching problem as in my case then this cost may be
necessary. (System.gc, System.gc in thread class below)

I made a very simple example below (compiles as-is :-) and similar to
my real code which has lots of threads. I think these general memory
usage concepts will be applicable to other areas and get the
OutOfMemoryError under control. Also changed startup setting for Java.

Thanks for the feedback,
TimJowers

package WebGen;
/**
* Sample memory usage for URLConnection
*/
import java.net.*;
import java.io.*;

public class Leaky {

    private URL url=null;
    private URLConnection urlConnection=null;
    private InputStream is=null;
    private StringBuffer sbuf = null;
    private byte[] bytesRead = new byte[4096];

    protected void zeroMem()
    {
        url=null;
        urlConnection=null;
        is=null;
        sbuf = null;
        bytesRead = null;
    }

    public boolean doWget ( String httpRequest ) throws Exception {

    System.out.println(httpRequest);

    try {
        url = new URL(httpRequest);
    } catch (MalformedURLException e) {
            throw new Exception("Malformed URL, must be
http://somehost[/file]");
    }
    try {
        urlConnection = url.openConnection();
    } catch (IOException e) {
            throw new Exception("Error opening URL Connection to " +
httpRequest);
    }

    DataOutputStream     printout;
    DataInputStream        input;

    urlConnection.setDoInput (true);
    urlConnection.setUseCaches (false);
    // may need to set header "Keep-Alive: close"
    urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
    urlConnection.setRequestProperty("Cache-Control","no-cache");

    if( !readHttp() )
    return false;

    closeAll();

    return true;
}

    class TestLeaky extends Thread
    {
        String httpRequest;
        public void run() {
            Leaky wget = new Leaky();
            try{
                wget.doWget( httpRequest );
                String result = wget.getOutput();
                System.out.println( httpRequest + " page has " + result.length() +
" characters" );
                wget.zeroMem(); // actually speeds gc
                wget=null;
                System.gc(); // probably overkill but best way to minimize memory
usage
                System.gc();
            } catch( Exception e) {
                System.err.println( e );
            }
        }
    }

    /** Main invoked from the prompt.
    * @param args Arguments from the Stdin
    * @throws Exception Throws Exception
    */
    static String[] urls = {"http://www.youart.net/",
"http://www.microsoft.net/",
                            "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
                            "http://www.nbc.com/", "http://www.abc.com/",
                            "http://www.cbs.com/", "http://www.bbc.com/",
                            "http://www.ebay.com/", "http://www.yahoo.com/",
                            "http://www.ncr.com/", "http://www.ibm.com/",
                            "http://www.ford.com/", "http://www.cisco.com/",
                            "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.ncr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.ebay.com/", "http://www.yahoo.com/",
        "http://www.googlr.com/", "http://www.ibm.com/",
        "http://www.ford.com/", "http://www.cisco.com/",
        "http://www.honda.com/", "http://www.toyota.com/",
        "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
        "http://www.nbc.com/", "http://www.abc.com/",
        "http://www.cbs.com/", "http://www.bbc.com/",
        "http://www.qbert.com/", "http://www.yahoo.com/",
        "http://www.starwars.com/", "http://www.nasa.gov/",
        "http://www.startrek.com/", "http://www.cisco.com/",
        "http://www.starman.com/", "http://www.mars.com/",
                            "http://www.cnn.com/"};
    public static void main ( String[] args ) throws Exception {

    Leaky lek = new Leaky();
    for( int e=0; e<urls.length; e++)
    {    TestLeaky thrd = lek.new TestLeaky();
        thrd.httpRequest = urls[e];
        thrd.start();
    }

    System.gc();
    System.gc();
    System.gc();
    // memory is leaked at this point. Next memory is magically returned
if we sleep awhile.
for( int s=0; s<60; s++ ) {
     try{ Thread.sleep(5000);}catch(Exception e){}
        System.gc();
}
}

    public Leaky() {    }

    private void closeAll() {
        if( urlConnection != null )
        try {
            urlConnection.getInputStream().close();
            if( urlConnection.getDoOutput() == true )
                urlConnection.getOutputStream().close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        urlConnection = null;
        url = null;
        is = null;
        //System.gc();
    }
    private boolean readHttp() throws Exception {
        try {
            sbuf = new StringBuffer();
            is = urlConnection.getInputStream();
            if (urlConnection.getHeaderField("Set-Cookie") != null) {
                    System.out.println(urlConnection.getHeaderField("Set-Cookie"));
                    sbuf.append(urlConnection.getHeaderField("Set-Cookie"));
            }
            int amount=0;
            System.out.println("Connected, retrieving file...");
            while ((amount = is.read(bytesRead)) > 0) {
                //System.out.println( "Just read bytes " + amount );
                    for (int i=0; i<amount; i++) {
                            sbuf.append((char)bytesRead[i]);
                    }
            }
        } catch (IOException e) {
            System.err.println( e );
        return false;
        }

        try {
            is.close();
        } catch (IOException e) {
                System.err.println("Error closing connection.");
            return false;
        }
        return true;
    }

    /** Returns the String returned by the Operation, like the content of
a Web page.
    * @return Returns a String object containing the received information
    */
    public String getOutput() {
        if( sbuf == null ) return "";
        String output = sbuf.toString();
        //sbuf.delete(0,sbuf.length());
        //sbuf = null;
        return output;
    }
}

Reply to this Message

Thomas Weidenfeller - 06 Jun 2005 08:41 GMT

> Thanks for everyone's feedback. I determined
> 1) closing input stream speeds up memory recovery (closeAll below)
> 2) setting references to null speeds up memory recovery (zeroMem below)
> 3) calling gc speeds up memory recovery. Of course, at a cost but if
> mem is the over-arching problem as in my case then this cost may be
> necessary. (System.gc, System.gc in thread class below)

2) is seldom needed (only if the method will hold onto the reference for
a long time after the object is needed). 3) is usually not needed at
all. The JVM will run the GC when it becomes low on memory. So when you
see an out of memory error, the JVM really has no memory, and did
previously try "everything" to gain free memory. A manual attempt to run
the GC is no different from a JVM's automatic attempt to run the GC.

Running the GC manually can not clean up any references you accidentally
hold on to. The GC can't fix coding errors. Get a memory profiler to
figure out who holds the references. Manually calling the GC just
needlessly burns CPU cycles.

/Thomas

Signature

The comp.lang.java.gui FAQ:
ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq

Reply to this Message

Chris Uppal - 06 Jun 2005 09:27 GMT

> 3) is usually not needed at
> all. The JVM will run the GC when it becomes low on memory. So when you
> see an out of memory error, the JVM really has no memory, and did
> previously try "everything" to gain free memory. A manual attempt to run
> the GC is no different from a JVM's automatic attempt to run the GC.

The point is that if some external resource ends up being managed (whether by
design or by accident) by finalisation (or reference queues, etc), then manual
GC may allow the finalisation process to 'see' the no-longer-needed resources
and clean them up /before/ the JVM realises that it needs to reclaim memory.
Not a good solution, and one that is never /guaranteed/ to work, but it can
help in some circumstances.

-- chris

Reply to this Message

Kevin McMurtrie - 04 Jun 2005 18:36 GMT

Your code isn't even close to complete. The interesting parts, like
what you do with the input stream, is missing.

Generally, temporary hangs and memory leaks are caused by not completing
the streams or abandoning HTTP 1.1 connections without specifying
"Keep-Alive: close" in the header.

Reply to this Message

URLConnection "memory leak"

Free Magazines