Java Forum / General / March 2006
Array Initial data gives code to large error
dcook@ccs-a.com - 27 Feb 2006 21:05 GMT Hello,
Wondering if someone may have a solution for my problem with Initial Data for object arrays. I have a 2D vector animator program that auto-generates data for Java code. All the vector objects data is output for a structures which I have converted to data classes. My problem is the data can get too large for Java and I get a "code to large" error from the compiler. So if the 2D project has something like 5,000 box vectors in it, it outputs 4 coordinate values top, left, bottom, right and a string for the box's name which is loaded into a box class that is part of a box class array of 5,000. At first, I tried something like this;
public class project_data { public static final int box_count = 5,000; public static box[] box_array = { new box(0,0,10,10,"box1"), new box(20,20,40,40,"box2"), ....}; }
I couldn't have very many boxes in the array before I got a "code to large" error so I then changed it to output the data as a string array that I can then parse into data which increased the amount of objects I can have. I created separated data classes for all data types. I had to make them string arrays because it is mixed data. Example;
public class box_data { public static final String[] data = { "0,0,10,10,\"box1\"", "20,20,40,40,\"box2\"", ...};
public static box[] box_data(int count) { box[] dobj = new box[count]; for (int i = 0; i < count; i++) { dobj = new box(data); } return dobj; } }
Class project_data { public static final int box_count = 5,000; public static box[] box_array = box_data(count); }
Is there another way I can define the data to get past this limit in Java or any suggestion of a better way so I can support more objects.
Thanks,
Dave
John C. Bollinger - 28 Feb 2006 02:04 GMT > Wondering if someone may have a solution for my problem with Initial > Data for object arrays. I have a 2D vector animator program that [quoted text clipped - 40 lines] > Is there another way I can define the data to get past this limit in > Java or any suggestion of a better way so I can support more objects. Write the object data to a file instead of generating Java code with it. Have your program read the file at startup and create the necessary objects. This will get around your code size problem very nicely, and will also solve the problem of needing to have a new program for every model.
 Signature John Bollinger jobollin@indiana.edu
James McGill - 28 Feb 2006 07:07 GMT > > public static final int box_count = 5,000; Please tell me that's a syntax error and not some confounded new feature.
dcook@ccs-a.com - 28 Feb 2006 18:51 GMT James,
Yes, this was a syntax error. It would actualy be;
public static final int box_count = 5000;
Dave
dcook@ccs-a.com - 28 Feb 2006 18:49 GMT John,
Thanks for the reply. This C++ animation app can output source in c or Java to allow customer to modify and customize. The main customer base is for c but see great potential for Java. Making the main C++ app output Java formated data object files is not an option at this point. Also, output all the data into a flat text file that I can parse into Class data would increase to an unresonable Java app startup time. There has to be a better solution.
Thanks,
Dave
Chris Uppal - 28 Feb 2006 19:15 GMT > Also, output all the data into a flat text file that I can parse into > Class data would increase to an unresonable Java app startup time. Why ? You already have text parsing code, and you are presumably happy with its performance. Loading a classfile with the same data embedded as Strings (if you were allowed to define that much data) would involve loading exactly the same data from disk as from your datafile, and what's more it involves loading it through a significantly more expensive code path.
> There has to be a better solution. Why ?
-- chris
dcook@ccs-a.com - 28 Feb 2006 20:12 GMT Chris,
Thanks for the reply. Well it is a lot less code to parse through a string array than reading from a flat file of unknown number of possible elements and class type elements. Also, if I try to use any special tags to mark elements and / or class data elements would complicate parsing since the data could contain one of these tags. So yes, it would greatly effect the startup time which is now already slow because of the parsing.
Thanks!
Dave
Chris Uppal - 01 Mar 2006 16:41 GMT > Thanks for the reply. Well it is a lot less code to parse through a > string array than reading from a flat file of unknown number of [quoted text clipped - 3 lines] > yes, it would greatly effect the startup time which is now already slow > because of the parsing. I think you are not understanding the way that Java (or the JVM) works. If you have a string in your program, then the route it takes to get into the runtime is complicated. It goes something like:
JVM finds classfile in some JAR. JVM decompresses classfile. JVM /parses/ the classfile to find methods, constants, strings, etc. ... lots of other stuff that's irrelevant here... JVM interns any strings. JVM executes code (from the classfile) which creates arrays, and then assigns String values to each element.
So, loading String data by including it in the Java source is just about the /slowest/ way you could possibly get that data into the runtime. Given that you have complete freedom over the datafile format, and that it's machine-generated so readability is presumably not a big issue, you should be able to load the data in a lot faster than the JVM can do it for you.
Java may look superficially like C++, but it is a completely different language with a totally different model of execution. You have to be prepared for the idea that you won't be able (unless you are lucky) to make a few small "tweaks" to an existing C++ code-generator and expect everything to work properly (although it would be easier than going in the opposite direction).
-- chris
dcook@ccs-a.com - 02 Mar 2006 19:51 GMT > you should be able to load the data in a lot faster than the JVM can do it for you. Chris,
Thanks for explaining this. From the way you explained this, it should be faster parsing my self than letting the JVM do it. I was assuming that the JVM was usually written in c and executes faster than me manually parsing using Java. Isn't that why it has System functions like System.arraycopy() because it runs faster at the JVM level than copying an array yourself in Java?
Thanks!
Dave
Chris Uppal - 03 Mar 2006 16:10 GMT > Thanks for explaining this. From the way you explained this, it should > be faster parsing my self than letting the JVM do it. I was assuming > that the JVM was usually written in c and executes faster than me > manually parsing using Java. The only real reason why parsing in Java might be noticeably slower than C (assuming you are running a modern JVM on a "normal" machine) is that you'd probably want to use Java Strings rather than raw data as byte[] arrays. That (unless you are working in UTF-16 anyway -- which I doubt) would involve doubling the amount of memory used, and hence taking longer to process. (BTW, the JVM has to perform an 8- to 16bit conversion when it's reading the Strings from the classfile too -- I forgot to mention that before).
I'd be inclined, as a first step, just to switch to some simple parsing, with a format defined to make parsing easy, and see how that goes. Or you could take Thomas Hawtin's suggestion of using serialisation as the on-disk form. In either case you can study the ./actual/ performance much better once there is code there to execute ;-) If the parsing time does turn out to be a major problem (which I rather doubt assuming the parser is sensibly defined and written) then I'd try using a binary format instead and probably use file-mapping to load the data in.
> Isn't that why it has System functions > like System.arraycopy() because it runs faster at the JVM level than > copying an array yourself in Java? Not really. It's more that (A) that routine was defined back in the days when Java was always interpreted. (B) there may still be interpretive JVMs (for resource-starved devices, perhaps). And (C) it was thought important to make that routine as fast as possible, and the way to do that is to include it in the platform, so that the implementation can use whatever "tricks" are appropriate to that platform's implementation without making users of the routine platform-dependent themselves.
-- chris
James McGill - 28 Feb 2006 07:27 GMT > Is there another way I can define the data to get past this limit in > Java or any suggestion of a better way so I can support more objects. Do you get further if you don't declare all your data "static?"
I'm guessing your JVM has to put all that static data into a "text" segment. (how this is done, is largely up to the JVM implementation.)
You might be able to tune your JVM, if you really want your program to work the way it's written. Or you might be able to put that data on the heap, and be a lot less crowded.
What I take from this, is that there is apparently some hard limit on class data, which means there's an effective limit on a class size. I've never run into this or even considered it before.
I have to assume you know what you're doing. What's the rationale for making all this data static, and does it still break if you make it volatile?
James
dcook@ccs-a.com - 28 Feb 2006 18:55 GMT James,
Thanks for the reply. Using the method of making the data a string array allows me the largest data elements. I have tryed it as volatile but was able to define larger amount of string arrays using static final.
Thanks!
Dave
James McGill - 28 Feb 2006 20:35 GMT > James, > > Thanks for the reply. Using the method of making the data a string > array allows me the largest data elements. I have tryed it as volatile > but was able to define larger amount of string arrays using static > final. I'm guessing that's because the static strings are static *references* to strings...
Cheers,
James
John C. Bollinger - 01 Mar 2006 04:29 GMT > What I take from this, is that there is apparently some hard limit on > class data, which means there's an effective limit on a class size. > I've never run into this or even considered it before. The class file format has a number of hard limits on the sizes of various parts. You can find all the gory details in the VM spec, http://java.sun.com/docs/books/vmspec/2nd-edition/html/ClassFile.doc.html Among them, each class's number of methods, number of fields, number of interfaces implemented, and number of constant pool entries is limited to 65534. The last is the biggest restriction (of those), as most class file substructures require one or more entries in the constant pool.
If you ever actually run into any of these limits with code written by hand then it will have long since been time to refactor or to completely redesign. Code-generating tools (such as the OP's) do sometimes hit one or another of the limits, however, which can be annoying.
 Signature John Bollinger jobollin@indiana.edu
Chris Uppal - 28 Feb 2006 09:30 GMT > Is there another way I can define the data to get past this limit in > Java or any suggestion of a better way so I can support more objects. I second John's advice. I just wanted to mention that the reason for this restriction is that initialisation expressions like the ones in your example are compiled into code. (The resulting bytecode looks just like a long series of assignments). There is a limit on the length of any method (which cannot be circumvented since it's built into the classfile format), so it -- for some unimaginable reason -- you /have/ to include the data in the code, then all you can do is split the initialisation into several sub-methods.
-- chris
Thomas Hawtin - 28 Feb 2006 15:42 GMT >> Is there another way I can define the data to get past this limit in >> Java or any suggestion of a better way so I can support more objects. > > I second John's advice. Yup. To get close to the generated code solution, I'd suggest using a resource rather than a loose file, and probably serialisation.
> I just wanted to mention that the reason for this > restriction is that initialisation expressions like the ones in your example [quoted text clipped - 3 lines] > unimaginable reason -- you /have/ to include the data in the code, then all you > can do is split the initialisation into several sub-methods. IIRC, you can exceed the limit. You just can't have debugging information or try/catch/finally/synchronized blocks in the extended area. Whether any particular compiler will generate such code is another matter.
JSR202 was supposed to increase various size limits. However, it does not appear to have done so. Perhaps huge JSPs from hell are out of fashion.
http://jcp.org/en/jsr/detail?id=202
Tom Hawtin
 Signature Unemployed English Java programmer http://jroller.com/page/tackline/
Chris Uppal - 28 Feb 2006 17:04 GMT > IIRC, you can exceed the limit. You just can't have debugging > information or try/catch/finally/synchronized blocks in the extended > area. Whether any particular compiler will generate such code is another > matter. The compiler would also have to generate wide goto's to supplement the branch instructions which only come in narrow flavours.
I thought it worth a giggle to try out with some hand-crafted bytecode. HelloWorld with 1e6 NOPs before the System.out.println(). The resulting classfile seems to be structurally OK (four different parsers manage to decode it), but the JVM throws it out with an error:
Exception in thread "main" java.lang.ClassFormatError: Invalid method Code length 1000009 in class file HelloWorld
Which is a pity, 'cos that would have been another solution for the OP -- generating bytecode directly instead of messing around with Java source ;-)
-- chris
dcook@ccs-a.com - 28 Feb 2006 19:04 GMT Chris,
Thanks for the reply. That is the same conclusion I came to but since this C++ Animation app outputs source code for c or Java and the main customer base is for c, I can not change it to break up the data into multiple sub-methods at this point. I think Java is a better solution for the customer base and if it catches on, this is the solution I plain to use. I may have to support mobile Java (midlets) before I can get the customer base I need to force a change in the C++ Animation app.
Thanks!
Dave
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|