 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / May 2006To store a huge table during start-up of a J2EE application
Hi all, Rather than making a db query all the time, I am considering the design of stroing a huge table( with 12 columns and may be about 500 rows) in some data structure , during start-up of my servlet. Memory is not a constraint, thats why we hare going for such an implementation. Please help for some questions below:- 1) Whats the best ADT or Data Structure for doing this? 2) In order to reduce the number of rows and ease manipulation, is it better to have an HttpBindinngListener which will strone the info about the user who had logged in and corresponding values/rows for that user? 3)Any other way ? Thanks for your time and advice. BEst regards, Ravi > Hi all, > [quoted text clipped - 11 lines] > > Thanks for your time and advice. Just a quibble about the wording of your subject line and question: I wouldn't call 500 rows of data with 12 columns of data in each row a "huge" table. I'm a database guy and we don't usually think a table is "huge" until it has MILLIONS of rows - or more. Now, if your columns are really wide, say 2 GB each or something like that, you are starting to get into slightly more significant territory but, generally speaking, I wouldn't call your table anything more than "small". As for the best design for what you are doing, I can't really suggest a good answer since I've never embedded a table of the kind you describe in a servlet. My servlets always get that sort of data from the database. Is this static data? In other words, is it ever going to change or is it just a lookup table of some kind, like the two-letter codes for the different states in the US? If the data is NOT static, I hope you give some thought to how you are going to keep it consistent with the source of the data, whether that is a database or a file. If the data can change, let's say it's a price table, you're going to run into BIG problems if the servlet version of the table has the old prices and you charge customers at the old prices when the new prices have gone into effect. -- Rhino -- Rhino Dear Rhino, I agree, the table is small as per your thoughts. Though for us, it is abig table. Yah, basically it is alook up kind of data. FOr example the role categories, the function ids, the action ids etc for a particular user. When the user logs in, we need to check the entitlement, whether this user is permitted to go for this function or not. Thats why we are caching it. Our priority is not for memory concumption, but execution time. Thats why we cant have a look up from servlet through an EJB or DAO kind of thing. Any thoughts to share with? After contemplating a bit, what I am thinking is this:- Assume that first four columns, col1...col4 are primary keys out of the total 12 columns. My search or lookup is always based on the primary key, i.e the first four columns. So I will have the startup init method in one of the servlets, where I will create two strings. First one is "col1__col2__col3__col4" (lets say the key)and second one is "col5:col6:col7:col8:...col12" (lets say the value).Then I will populate a hashtable with key and value. Whenever a search is needed, I would first construct the key and look for it.Once I get the value, I will STringTokenize it and use it wherever required. In case I need a search for only col1( may be in future), I would have another hashtable to maintain col1_...col4. Is this fine? Please comment. Thanks Best regards, Ravi >> Hi all, >> [quoted text clipped - 38 lines] > -- > Rhino > Dear Rhino, > [quoted text clipped - 25 lines] > > Is this fine? Please comment. Thanks The first thing to do is to create an Object that encapsulates these 12 columns i.e. attributes of that row. Second this is (assuming for a second that your keying col1...col4 key is good enough), store the "object" that you created in the previous step as value of the hashtable - dont get into the business of stringtokenizing etc, they are costly and round-about way of doing the things at hand. Now to your actual problem, if your primary keys and index are setup correctly, a database query should not run for more than tens of milliseconds - granted that there are some exceptions too. Since you seem to indicate user-specific data, dont see the value of losing the ability to do SQL like queries in return for 10 millis of time. And actually, when you scale up with number of users, the memory will kinda get abused, but the database can easily scale up to a high concurrent query load. -- Arvind ... > Any thoughts to share with? After contemplating a bit, what I am thinking is > this:- Assume that first four columns, col1...col4 are primary keys out of [quoted text clipped - 4 lines] > "col5:col6:col7:col8:...col12" (lets say the value).Then I will populate a > hashtable with key and value. ... I don't recommend encoding as String and tokenizing. It makes everything quite unnecessarily complicated. You would need to either impose a prohibition on "_" as data, or do some quoting. It would be simpler to create classes for the keys and values. You would need to write good, consistent, equals and hashCode methods for the key class. Also, don't ignore the database idea. It scales to almost any size, and minimizes the amount of change in your program that is required to support additional look-up methods. Patricia > Rather than making a db query all the time, I am considering the design of >stroing a huge table( with 12 columns and may be about 500 rows) in some [quoted text clipped - 3 lines] > >1) Whats the best ADT or Data Structure for doing this? Typically, that's one collection (for table) stored inside the other collection (for record). But it really depends on what you need to do with that table. If you're always fetching a record on several criteria, you might want to use List inside the List. If you have one key column, and are always accessing data through that column? That's Map, where you store key column in key, and List in value. Or if your access patterns are more complicated, you might want to roll your own classes. Use one class with getters and setters for record, and than store that records in some sort of list. Or in other custom class which uses list to store records, but has custom methods which access the list. If your table is not read-only, you might want to have custom add/put/set method which writes record to both in-memory table and to the database. Or have some sort of queue where you're marking which records were added or need updating, and fork another thread to empty that queue. If several threads need to access the table, watch for thread safety. Domchi  SignatureOuroboros ltd. - http://www.ouroboros.hr Antispam: to reply, remove extra monkey from reply-to address. >Hi all, > [quoted text clipped - 9 lines] >who had logged in and corresponding values/rows for that user? >3)Any other way ? Perhaps you can have the best of both worlds by using an in-memory database such as <http://hsqldb.org/>. Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.