> Your application may be exceeding the wait_timeout server setting, as
> described in the manual:
>
> http://dev.mysql.com/doc/refman/5.0/en/instance-manager-command-options.html
The setting is 8 hours, and mine is still set to that default, and the
program
stops after only 42 minutes or ~100,000 rows, so I dont think thats the
case.
> However, I'm tempted to ask whether you *really* need to retrieve all 9
> million rows?
Yes, it is going to be a program which runs regularly which transfers the
contents of a backup
MySQL database to a Lucene index, so yeah, the entire data set needs to be
read.
Whats really puzzling me is that I have been using Perl DBI on this table
for years in production and never ran into this issue.
I also have been seeing a recent surge in forum posts on the MySQL forums
for jdb/jconnection people with the same error
with no solution which has worked for me.
I realize I might be able to get around this problem by hacking the query
into LIMIT statements etc but I would much rather
solve the problem then work around it. This is not expected MySQL behavior
so either I am doing something wrong or a
recent update to MySQL/JDBC/jconnect/jdk/jre has a bug.
joeNOSPAM@BEA.com - 25 May 2007 20:47 GMT
> I realize I might be able to get around this problem by hacking the query
> into LIMIT statements etc but I would much rather
> solve the problem then work around it. This is not expected MySQL behavior
> so either I am doing something wrong or a
> recent update to MySQL/JDBC/jconnect/jdk/jre has a bug.
This may be your client JVM getting progressively slower, perhaps
due to memory limitations, and having to scrabble for free memory
as you process your result set. Try:
1 - Starting the JVM with arguments to give it more memory.
2 - Make sure you get forward-only, non-scrollable result sets,
and close every JDBC object ASAP.
3 - Try a plain statement for the query, not a prepared one. Unless
you're going to rerun the same prepared statement a lot of times,
there may be no benefit and some loss in using them.
Joe Weinstein at BEA Systems
David Harper - 25 May 2007 21:40 GMT
[SNIP]
>> However, I'm tempted to ask whether you *really* need to retrieve all 9
>> million rows?
[quoted text clipped - 3 lines]
> MySQL database to a Lucene index, so yeah, the entire data set needs to
> be read.
According to the Lucene web site (http://lucene.apache.org/java/docs/),
"Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for nearly
any application that requires full-text search, especially cross-platform."
MySQL already provides full-text search so I can't help but think that
you may not be making optimal use of the software components at hand.
Given that you need to store your text in a database, why not make use
of the full-text search functionality provided by MySQL?
David Harper
Cambridge, England
seth brundle - 26 May 2007 01:58 GMT
> "Apache Lucene is a high-performance, full-featured text search engine
> library written entirely in Java.
> .... why not make use of the full-text search functionality provided by
> MySQL?
I've been using MySQL fulltext for user web search on this dataset for about
4 years, but the requirements and dataset have outgrown MySQL's limited
search capabilties.