Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / February 2006

Tip: Looking for answers? Try searching our database.

Time - BuffererdReader takes read in data - store in hash Map TAKING FOREVER

Thread view: 
PythonAnimal@gmail.com - 03 Feb 2006 00:55 GMT
I am having prolems, I am using bufferedReader to read in data from a
file store it in a hashtabl then store it in a hashmap.  My problem is
even with threading it is running very slow, 45 seconds to load 15,000
lines.  Is this accurate or is my code off?
Here is a snippet.  Any help would be appericated

while(line != null)
{

    while(headingToken.hasMoreTokens())
    {    counter++;
        heading = headingToken.nextToken();
        try{
        data = lineToken.nextToken();
        //System.out.print(data + " " );
    }catch(Exception e) {    data = "";                }
        if(heading.equals(""))
        heading += "heading";
                                heading += counter + "NoHeading";
        _headingColumnsOutputFile += heading + ",";

        genevaDataRows.put(heading,data);
    }

/************* in my code putting and removing

    String bifurcationPrimarykey =
genevaDataRows.get(_primaryKey).toString() +
genevaDataRows.get("Description").toString() + genevaDataRows.get("Tax
Lot");
    genevaDataRows.put("Primary Key",bifurcationPrimarykey);
    this._genevaTradesMap.put (bifurcationPrimarykey,genevaDataRows);
/****************
    line = bufRea.readLine();
    if(line != null)
    lineToken = new HBMJTokenizer2(line, ",", "\"");
    headingToken = new HBMJTokenizer2(_dataHeading, ",", "\"");
    genevaDataRows = new  Hashtable();        // resets hashtable so another row
may be inputted
    counter = 0;
    nd2 = new Date();
    System.out.println("End line != null " + this._accountingPeriod + " "
+ nd2.getMinutes() + " " + nd2.getSeconds());
}
Seems it should not take this long!!
thanks gfor any advice
Kevin - 03 Feb 2006 02:23 GMT
1) did you give the buffer reader a proper (large) buffer?
2) adding Strings is very slow. Repalce with StringBuffer will decrease
the time by 100 times in one of my code.
Roedy Green - 03 Feb 2006 11:43 GMT
>My problem is
>even with threading it is running very slow, 45 seconds to load 15,000
>lines.  Is this accurate or is my code off?

For super speed read the entire file into ram as an array of bytes,
then convert to string, then parse it yourself. Here is the code to do
it.  This way, I process 200 files a second.

The other way to do it, is to crank your BufferedReader  buffer up to
say 64K.

package com.mindprod.hunkio;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.UnsupportedEncodingException;

/**
* Read and write entire files as Strings. Presumes the default
encoding. No
* main method.
*
* @author Roedy Green <br>
*         version 1.5 - 2006-01-25 - readFile/writeFile with
non-default
*         encoding, raw read/write entire file. <br>
*         version 1.4 - 2006-01-14 - readEntire file now works with
multi-byte
*         encodings. <br>
*         version 1.3 - 2005-06-18 add more consistent bat files.<br>
* @since 2002 July 12
*/
public class HunkIO {

   /**
    * Create a temporary file, Slightly smarter version of
File.createTempFile
    *
    * @param prefix
    *        beginning letters of filename
    * @param suffix
    *        ending letters of filename.
    * @param near
    *        directory where to put file, or file to place this temp
file near
    *        in the same directory. null means put the temp file in
the current
    *        directory.
    * @return A temporary file. It will not automatically delete on
program
    *         completion, however.
    * @exception IOException
    */
   public static File createTempFile ( String prefix, String suffix,
File near )
       throws IOException
       {
       if ( near != null )
           {
           if ( near.isDirectory() )
               {
               return File.createTempFile( prefix, suffix, near );
               }
           else if ( near.isFile() )
               {
               String parent = near.getParent();
               if ( parent != null )
                   {
                   File dir = new File( parent );
                   if ( dir.isDirectory() )
                       {
                       return File.createTempFile( prefix, suffix,
dir );
                       }
                   }
               }
           }
       // anything else, just create in the current directory.
       return File.createTempFile( prefix, suffix );
       }

   /**
    * read file of bytes without conversion
    *
    * @param fromFile
    *        file to read
    * @return byte array representing whole file
    * @throws IOException
    */
   public static byte[] rawReadEntireFile ( File fromFile ) throws
IOException
       {
       int size = (int)fromFile.length();

       FileInputStream fis = new FileInputStream( fromFile );
       // R E A D
       byte[] rawContents = new byte[ size ];
       int bytesRead = fis.read( rawContents );
       if ( bytesRead != size )
           {
           throw new IOException( "error: problems reading file " +
fromFile );
           }
       // C L O S E
       fis.close();
       return rawContents;
       }

   /**
    * read file of bytes without conversion
    *
    * @param fromFile
    *        file to read
    * @return byte array representing whole file
    * @throws IOException
    */
   public static byte[] rawReadEntireFile ( String fromFile )
       throws IOException
       {
       return rawReadEntireFile( new File( fromFile ) );
       }

   /**
    * Get all text in a file.
    *
    * @param fromFile
    *        file where to get the text.
    * @return all the text in the File.
    * @exception IOException
    */
   public static String readEntireFile ( File fromFile ) throws
IOException
       {
       // decode with default encoding and return entire file as one
big string
       return new String( rawReadEntireFile( fromFile ) );
       } // end readEntireFile

   /**
    * Get all text in a file.
    *
    * @param fromFile
    *        file where to get the text.
    * @param encoding
    *        name of the encoding to use to translate the bytes in
the file
    *        into chars e.g. "windows-1252" "UTF-8" "UTF-16"
    * @return all the text in the File.
    * @exception IOException
    * @exception UnsupportedEncodingException
    *            If the named encoding is not supported
    */
   public static String readEntireFile ( File fromFile, String
encoding )
       throws IOException, UnsupportedEncodingException
       {
       // decode with encoding and return entire file as one big
string
       return new String( rawReadEntireFile( fromFile ), encoding );
       } // end readEntireFile

   /**
    * Get all text in a file.
    *
    * @param fromFile
    *        Name of file where to get the text.
    * @return all the text in the File.
    * @exception IOException
    */
   public static String readEntireFile ( String fromFile ) throws
IOException
       {
       return readEntireFile( new File( fromFile ) );
       } // end readEntireFile

   /**
    * Get all text in a file.
    *
    * @param fromFile
    *        Name of file where to get the text.
    * @param encoding
    *        name of the encoding to use to translate the bytes in
the file
    *        into chars e.g. "windows-1252" "UTF-8" "UTF-16"
    * @return all the text in the File.
    * @exception IOException
    * @exception UnsupportedEncodingException
    *            If the named encoding is not supported
    */
   public static String readEntireFile ( String fromFile, String
encoding )
       throws IOException, UnsupportedEncodingException
       {
       return readEntireFile( new File( fromFile ), encoding );
       } // end readEntireFile

   /**
    * Write all the text in a file
    *
    * @param toFile
    *        file where to write the text
    * @param text
    *        Text to write
    * @exception IOException
    */
   public static void writeEntireFile ( File toFile, String text )
       throws IOException
       {
       // default encoding
       FileWriter fw = new FileWriter( toFile );
       // W R I T E
       fw.write( text );
       // C L O S E
       fw.close();
       } // end writeEntireFile

   /**
    * Write all the text in a file
    *
    * @param toFile
    *        file where to write the text
    * @param text
    *        Text to write
    * @param encoding
    *        name of the encoding to use to translate chars to bytes
e.g.
    *        "windows-1252" "UTF-8" "UTF-16"
    * @exception IOException
    * @exception UnsupportedEncodingException
    *            If the named encoding is not supported
    */
   public static void writeEntireFile ( File toFile, String text,
       String encoding ) throws IOException
       {
       // supplied encoding
       FileOutputStream fos = new FileOutputStream( toFile );
       OutputStreamWriter osw = new OutputStreamWriter( fos, encoding
);
       // W R I T E
       osw.write( text );
       // C L O S E
       osw.close();
       } // end writeEntireFile

   /**
    * Write all the text in a file
    *
    * @param toFile
    *        Name of file where to write the text
    * @param text
    *        Text to write
    * @exception IOException
    */
   public static void writeEntireFile ( String toFile, String text )
       throws IOException
       {
       writeEntireFile( new File( toFile ), text );

       } // end writeEntireFile

   /**
    * Write all the text in a file
    *
    * @param toFile
    *        file where to write the text
    * @param text
    *        Text to write
    * @param encoding
    *        name of the encoding to use to translate chars to bytes
e.g.
    *        "windows-1252" "UTF-8" "UTF-16"
    * @exception IOException
    * @exception UnsupportedEncodingException
    *            If the named encoding is not supported
    */
   public static void writeEntireFile ( String toFile, String text,
       String encoding ) throws IOException
       {
       writeEntireFile( new File( toFile ), text, encoding );
       } // end writeEntireFile

} // end HunkIO

Signature

Canadian Mind Products, Roedy Green.
http://mindprod.com Java custom programming, consulting and coaching.



Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.