Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / April 2007

Tip: Looking for answers? Try searching our database.

how to reach end of file?

Thread view: 
ruds - 25 Feb 2007 04:34 GMT
Hello,
I'm reading a file which is sbout 3.5 GB in size. And extracting some
fields from the file.
For smaller file the program works fine but for this file , it is not
reaching the end of file and exits.
How should I force the program to reach end of file?

To read the file I'm using:
while((str=br.readLine())!=null)
{
 //set of operations
}

My sample code is:

public void getLine(String file) throws IOException
    {
        fr=new FileReader(file);
        br=new BufferedReader(fr);
        no=0;
        i=0;
        j=0;
        while((str=br.readLine())!=null)
        {
            no++;

            if(str.startsWith("$SUBCASE"))
            {
                st1= new StringTokenizer(str," ");
                while(st1.hasMoreTokens())
                {
                    tok1=st1.nextToken();
                    if(tok1.equals("="))
                    {
                        tok2=st1.nextToken();
                        if(tok2.equals(id))
                            continue;
                        else
                        {
                            id=tok2;
                            subcase[j]=Integer.parseInt(id);
                            line[i]=no-5;
                            System.out.println("Subcase="+subcase[j]);
                            i++;
                            j++;

                        }
                    }
                }
            }
            else
                continue;
        }
        System.out.println("end of getLine");
        fr.close();

    }

Please help ! Its urgent.
Andrew Thompson - 25 Feb 2007 04:56 GMT
...
> I'm reading a file which is sbout 3.5 GB in size. And extracting some
> fields from the file.
> For smaller file the program works fine but for this file , it is not
> reaching the end of file and exits.

And what exactly is the error output?
(* Or would you like us to simply guess?)

> How should I force the program to reach end of file?

Hit it with a stick?

* Maybe it is just lazy.

> To read the file I'm using:
..
> My sample code is:

I term that a sample 'snippet', and always
recommend (as opposed to code snippets)
compilable, runnable code.

..
> Please help ! Its urgent.

Pay me, then I might care about your schedule.

Andrew T.
ruds - 25 Feb 2007 06:40 GMT
It does not give any error...
It just exits the program before printing "end of getLine".
Andrew Thompson - 25 Feb 2007 06:53 GMT
> It does not give any error...

I am surprised.

> It just exits the program before printing "end of getLine".

..hmm.  Can you supply an SSCCE that does the
same*?
<http://www.physci.org/codes/sscce.html>

I strongly suspect this is an OutOfMemoryError
(this 3.5 Gb file goes somewhat beyond the standard
64 Meg. assigned to a regular Java app.), but
your stated results do not support that (no OOME
stacktrace), perhaps I am mistaken.

* it would be very handy if it could also
generate a File of approx. 3.5 Gig, before
attempting to read that file, and failing.

Andrew T.
Patricia Shanahan - 25 Feb 2007 15:15 GMT
>> It does not give any error...
>
[quoted text clipped - 15 lines]
> generate a File of approx. 3.5 Gig, before
> attempting to read that file, and failing.

One of my Java programs reads a 6,933,389,378 byte file. I can tell it
reads the whole file, because it generates two output files that each
have one line per line of the input, and the line counts match.

Here are a couple of possibilities:

1. A logic error in the program that is triggered by some condition that
exists in the full file but not in shorter excerpts.

2. A catch statement somewhere in the program that is concealing a
condition such as running out of heap space.

Patricia
Arne Vajhøj - 25 Feb 2007 16:14 GMT
> I strongly suspect this is an OutOfMemoryError
> (this 3.5 Gb file goes somewhat beyond the standard
> 64 Meg. assigned to a regular Java app.), but
> your stated results do not support that (no OOME
> stacktrace), perhaps I am mistaken.

A while loop with a BufferedReader readLine should
not read the entire file into memory. Going through
a 3.5 GB file with just 64 MB of heap space should
not be a problem.

Arne
Nigel Wade - 26 Feb 2007 10:26 GMT
> It does not give any error...
> It just exits the program before printing "end of getLine".

What do you do with IOException?

Does the JVM/environment in which you are operating support files of this size?
Some 32bit environments are limited to 2GB files.

Signature

Nigel Wade, System Administrator, Space Plasma Physics Group,
           University of Leicester, Leicester, LE1 7RH, UK
E-mail :    nmw@ion.le.ac.uk
Phone :     +44 (0)116 2523548, Fax : +44 (0)116 2523555

su_dang@hotmail.com - 26 Feb 2007 17:28 GMT
> > It does not give any error...
> > It just exits the program before printing "end of getLine".
[quoted text clipped - 9 lines]
> E-mail :    n...@ion.le.ac.uk
> Phone :     +44 (0)116 2523548, Fax : +44 (0)116 2523555

Not sure if this is a problem or not, but you might have an overflow
problem.  The maximum value of an int is 2,147,483,647, but your file
is a lot bigger than that.  If you are using int to count something in
your program, try to change it to long instead.
ruds - 07 Apr 2007 04:37 GMT
I'm executing the program on AIX OS which as per my information does
support huge files(larger than 2GB)..
Please tell me, How do i detect what actual problem is there out of
all suggested ones ???


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.