Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / March 2006

Tip: Looking for answers? Try searching our database.

Array Initial data gives code to large error

Thread view: 
dcook@ccs-a.com - 27 Feb 2006 21:05 GMT
Hello,

Wondering if someone may have a solution for my problem with Initial
Data for object arrays. I have a 2D vector animator program that
auto-generates data for Java code. All the vector objects data is
output for a structures which I have converted to data classes. My
problem is the data can get too large for Java and I get a "code to
large" error from the compiler. So if the 2D project has something
like 5,000 box vectors in it, it outputs 4 coordinate values top, left,
bottom, right and a string for the box's name which is loaded into a
box class that is part of a box class array of 5,000. At first, I tried
something like this;

public class project_data {
public static final int box_count = 5,000;
public static box[] box_array = { new box(0,0,10,10,"box1"), new
box(20,20,40,40,"box2"),   ....};
}

I couldn't have very many boxes in the array before I got a "code
to large" error so I then changed it to output the data as a string
array that I can then parse into data which increased the amount of
objects I can have. I created separated data classes for all data
types. I had to make them string arrays because it is mixed data.
Example;

public class box_data {
public static final String[] data = { "0,0,10,10,\"box1\"",
"20,20,40,40,\"box2\"", ...};

public static box[] box_data(int count) {
box[] dobj = new box[count];
for (int i = 0; i < count; i++) {
dobj = new box(data);
}
return dobj;
}
}

Class project_data {
public static final int box_count = 5,000;
public static box[] box_array = box_data(count);
}

Is there another way I can define the data to get past this limit in
Java or any suggestion of a better way so I can support more objects.

Thanks,

Dave
John C. Bollinger - 28 Feb 2006 02:04 GMT
> Wondering if someone may have a solution for my problem with Initial
> Data for object arrays. I have a 2D vector animator program that
[quoted text clipped - 40 lines]
> Is there another way I can define the data to get past this limit in
> Java or any suggestion of a better way so I can support more objects.

Write the object data to a file instead of generating Java code with it.
 Have your program read the file at startup and create the necessary
objects.  This will get around your code size problem very nicely, and
will also solve the problem of needing to have a new program for every
model.

Signature

John Bollinger
jobollin@indiana.edu

James McGill - 28 Feb 2006 07:07 GMT
> > public static final int box_count = 5,000;

Please tell me that's a syntax error and not some confounded
new feature.
dcook@ccs-a.com - 28 Feb 2006 18:51 GMT
James,

Yes, this was a syntax error.  It would actualy be;

public static final int box_count = 5000;

Dave
dcook@ccs-a.com - 28 Feb 2006 18:49 GMT
John,

Thanks for the reply.  This C++ animation app can output source in c or
Java to allow customer to modify and customize.  The main customer base
is for c but see great potential for Java.  Making the main C++ app
output Java formated data object files is not an option at this point.
Also, output all the data into a flat text file that I can parse into
Class data would increase to an unresonable Java app startup time.
There has to be a better solution.

Thanks,

Dave
Chris Uppal - 28 Feb 2006 19:15 GMT
> Also, output all the data into a flat text file that I can parse into
> Class data would increase to an unresonable Java app startup time.

Why ?  You already have text parsing code, and you are presumably happy with
its performance.  Loading a classfile with the same data embedded as Strings
(if you were allowed to define that much data) would involve loading exactly
the same data from disk as from your datafile, and what's more it involves
loading it through a significantly more expensive code path.

> There has to be a better solution.

Why ?

   -- chris
dcook@ccs-a.com - 28 Feb 2006 20:12 GMT
Chris,

Thanks for the reply.  Well it is a lot less code to parse through a
string array than reading from a flat file of unknown number of
possible elements and class type elements.  Also, if I try to use any
special tags to mark elements and / or class data elements would
complicate parsing since the data could contain one of these tags.  So
yes, it would greatly effect the startup time which is now already slow
because of the parsing.

Thanks!

Dave
Chris Uppal - 01 Mar 2006 16:41 GMT
> Thanks for the reply.  Well it is a lot less code to parse through a
> string array than reading from a flat file of unknown number of
[quoted text clipped - 3 lines]
> yes, it would greatly effect the startup time which is now already slow
> because of the parsing.

I think you are not understanding the way that Java (or the JVM) works.  If you
have a string in your program, then the route it takes to get into the runtime
is complicated.  It goes something like:

   JVM finds classfile in some JAR.
   JVM decompresses classfile.
   JVM /parses/ the classfile to find methods, constants, strings, etc.
   ... lots of other stuff that's irrelevant here...
   JVM interns any strings.
   JVM executes code (from the classfile) which creates arrays, and then
   assigns String values to each element.

So, loading String data by including it in the Java source is just about the
/slowest/ way you could possibly get that data into the runtime.  Given that
you have complete freedom over the datafile format, and that it's
machine-generated so readability is presumably not a big issue, you should be
able to load the data in a lot faster than the JVM can do it for you.

Java may look superficially like C++, but it is a completely different language
with a totally different model of execution.  You have to be prepared for the
idea that you won't be able (unless you are lucky) to make a few small "tweaks"
to an existing C++ code-generator and expect everything to work properly
(although it would be easier than going in the opposite direction).

   -- chris
dcook@ccs-a.com - 02 Mar 2006 19:51 GMT
> you should be able to load the data in a lot faster than the JVM can do it for you.

Chris,

Thanks for explaining this.  From the way you explained this, it should
be faster parsing my self than letting the JVM do it.  I was assuming
that the JVM was usually written in c and executes faster than me
manually parsing using Java.  Isn't that why it has System functions
like System.arraycopy() because it runs faster at the JVM level than
copying an array yourself in Java?

Thanks!

Dave
Chris Uppal - 03 Mar 2006 16:10 GMT
> Thanks for explaining this.  From the way you explained this, it should
> be faster parsing my self than letting the JVM do it.  I was assuming
> that the JVM was usually written in c and executes faster than me
> manually parsing using Java.

The only real reason why parsing in Java might be noticeably slower than C
(assuming you are running a modern JVM on a "normal" machine) is that you'd
probably want to use Java Strings rather than raw data as byte[] arrays.  That
(unless you are working in UTF-16 anyway -- which I doubt) would involve
doubling the amount of memory used, and hence taking longer to process.  (BTW,
the JVM has to perform an 8- to 16bit conversion when it's reading the Strings
from the classfile too -- I forgot to mention that before).

I'd be inclined, as a first step, just to switch to some simple parsing, with a
format defined to make parsing easy, and see how that goes.  Or you could take
Thomas Hawtin's suggestion of using serialisation as the on-disk form.  In
either case you can study the ./actual/ performance much better once there is
code there to execute ;-)  If the parsing time does turn out to be a major
problem (which I rather doubt assuming the parser is sensibly defined and
written) then I'd try using a binary format instead and probably use
file-mapping to load the data in.

> Isn't that why it has System functions
> like System.arraycopy() because it runs faster at the JVM level than
> copying an array yourself in Java?

Not really.  It's more that (A) that routine was defined back in the days when
Java was always interpreted.  (B) there may still be interpretive JVMs (for
resource-starved devices, perhaps).  And (C) it was thought important to make
that routine as fast as possible, and the way to do that is to include it in
the platform, so that the implementation can use whatever "tricks" are
appropriate to that platform's implementation without making users of the
routine platform-dependent themselves.

   -- chris
James McGill - 28 Feb 2006 07:27 GMT
> Is there another way I can define the data to get past this limit in
> Java or any suggestion of a better way so I can support more objects.

Do you get further if you don't declare all your data "static?"

I'm guessing your JVM has to put all that static data into a "text"
segment. (how this is done, is largely up to the JVM implementation.)

You might be able to tune your JVM, if you really want your program to
work the way it's written.  Or you might be able to put that data on the
heap, and be a lot less crowded.  

What I take from this, is that there is apparently some hard limit on
class data, which means there's an effective limit on a class size.
I've never run into this or even considered it before.  

I have to assume you know what you're doing.  What's the rationale for
making all this data static, and does it still break if you make it
volatile?

James
dcook@ccs-a.com - 28 Feb 2006 18:55 GMT
James,

Thanks for the reply.  Using the method of making the data a string
array allows me the largest data elements.  I have tryed it as volatile
but was able to define larger amount of string arrays using static
final.

Thanks!

Dave
James McGill - 28 Feb 2006 20:35 GMT
> James,
>
> Thanks for the reply.  Using the method of making the data a string
> array allows me the largest data elements.  I have tryed it as volatile
> but was able to define larger amount of string arrays using static
> final.

I'm guessing that's because the static strings are static *references*
to strings...  

Cheers,

James
John C. Bollinger - 01 Mar 2006 04:29 GMT
> What I take from this, is that there is apparently some hard limit on
> class data, which means there's an effective limit on a class size.
> I've never run into this or even considered it before.  

The class file format has a number of hard limits on the sizes of
various parts.  You can find all the gory details in the VM spec,
http://java.sun.com/docs/books/vmspec/2nd-edition/html/ClassFile.doc.html
Among them, each class's number of methods, number of fields, number of
interfaces implemented, and number of constant pool entries is limited
to 65534.  The last is the biggest restriction (of those), as most class
file substructures require one or more entries in the constant pool.

If you ever actually run into any of these limits with code written by
hand then it will have long since been time to refactor or to completely
redesign.  Code-generating tools (such as the OP's) do sometimes hit one
or another of the limits, however, which can be annoying.

Signature

John Bollinger
jobollin@indiana.edu

Chris Uppal - 28 Feb 2006 09:30 GMT
> Is there another way I can define the data to get past this limit in
> Java or any suggestion of a better way so I can support more objects.

I second John's advice.  I just wanted to mention that the reason for this
restriction is that initialisation expressions like the ones in your example
are compiled into code. (The resulting bytecode looks just like a long series
of assignments).  There is a limit on the length of any method (which cannot be
circumvented since it's built into the classfile format), so it -- for some
unimaginable reason -- you /have/ to include the data in the code, then all you
can do is split the initialisation into several sub-methods.

   -- chris
Thomas Hawtin - 28 Feb 2006 15:42 GMT
>> Is there another way I can define the data to get past this limit in
>> Java or any suggestion of a better way so I can support more objects.
>
> I second John's advice.

Yup. To get close to the generated code solution, I'd suggest using a
resource rather than a loose file, and probably serialisation.

>                          I just wanted to mention that the reason for this
> restriction is that initialisation expressions like the ones in your example
[quoted text clipped - 3 lines]
> unimaginable reason -- you /have/ to include the data in the code, then all you
> can do is split the initialisation into several sub-methods.

IIRC, you can exceed the limit. You just can't have debugging
information or try/catch/finally/synchronized blocks in the extended
area. Whether any particular compiler will generate such code is another
matter.

JSR202 was supposed to increase various size limits. However, it does
not appear to have done so. Perhaps huge JSPs from hell are out of fashion.

http://jcp.org/en/jsr/detail?id=202

Tom Hawtin
Signature

Unemployed English Java programmer
http://jroller.com/page/tackline/

Chris Uppal - 28 Feb 2006 17:04 GMT
> IIRC, you can exceed the limit. You just can't have debugging
> information or try/catch/finally/synchronized blocks in the extended
> area. Whether any particular compiler will generate such code is another
> matter.

The compiler would also have to generate wide goto's to supplement the branch
instructions which only come in narrow flavours.

I thought it worth a giggle to try out with some hand-crafted bytecode.
HelloWorld with 1e6 NOPs before the System.out.println().  The resulting
classfile seems to be structurally OK (four different parsers manage to decode
it), but the JVM throws it out with an error:

   Exception in thread "main" java.lang.ClassFormatError:
   Invalid method Code length 1000009 in class file HelloWorld

Which is a pity, 'cos that would have been another solution for the OP --
generating bytecode directly instead of messing around with Java source ;-)

   -- chris
dcook@ccs-a.com - 28 Feb 2006 19:04 GMT
Chris,

Thanks for the reply.  That is the same conclusion I came to but since
this C++ Animation app outputs source code for c or Java and the main
customer base is for c, I can not change it to break up the data into
multiple sub-methods at this point.  I think Java is a better solution
for the customer base and if it catches on, this is the solution I
plain to use.  I may have to support mobile Java (midlets) before I can
get the customer base I need to force a change in the C++ Animation
app.

Thanks!

Dave


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.