Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsWhite Papers
Discussion GroupsFirst AidDatabasesJavaBeansGUIJava 3DVirtual MachineCORBASecurityToolsGeneral
Java DirectoryOpen Source ProjectsSample Book ChaptersUser GroupsWeb Resources
Related Topics
Databases.NETMore Topics ...

Java Forum / General / November 2005

Tip: Looking for answers? Try searching our database.

Application log file parser

Thread view: 
Ben_ - 25 Jun 2004 21:36 GMT
Hello,

Are there tools to parse application log files (typically stdout & stderr,
but also Log4J output) in a smart way ?

Basically, I'd like to parse them and compute stats like when, how many
times and how often exceptions of a class have been raised.

Format is predictable, because we'll look for well known patterns, but there
may be so many different log statements to search for that I'm looking for a
flexible and productive way of doing.

Thanks.
scott - 26 Jun 2004 05:50 GMT
The the new version of log4j's Chainsaw, configured to use a
LogFilePatternReceiver, will do exactly this.

LogFilePatternReceiver can parse events from a log file based on a
pattern you provide, and can 'tail' the file as well.

Chainsaw also provides a simple expression syntax which can be used to
dynamically filter, find and colorize events.

Here's a link to Chainsaw V2.  It's available via Web Start:
http://logging.apache.org/log4j/docs/chainsaw.html

Here's a link to the LogFilePatternReceiver (read the javadoc for
information on how to specify the pattern in your file and the
'tailing' parameter):
http://cvs.apache.org/viewcvs.cgi/logging-log4j/src/java/org/apache/log4j/varia/
LogFilePatternReceiver.java?rev=1.12&view=auto


If you have questions, feel free to send them to the log4j-user
mailing list.

Scott
sdeboy@apache.org

Configure Chainsaw to use a LogFilePatternReceiver
> Hello,
>
[quoted text clipped - 9 lines]
>
> Thanks.
Ben_ - 28 Jun 2004 06:06 GMT
I'll check the LogFilePatternReceiver.

I think LogParser from Microsoft
(http://www.microsoft.com/downloads/details.aspx?FamilyID=8cde4028-e247-45be
-bab9-ac851fc166a4&DisplayLang=en) implements pretty much what I have in
mind: it searches files for named-fields with SQL-like syntax. What's
missing is the ability to search any file formats (it's targeted at http
logs) by allowing to define new named-fields.
Craig - 28 Jun 2004 21:36 GMT
Ok Scott/All, you got me interested.  I downloaded Chainsaw, but the
instructions on creating receivers is almost non-existant.  Here's
what I want to do:

1) Run a Java app which creates a log file on a Unix server somewhere.
The Java app does not use log4j to create entries -- its older, and
uses its own internally defined class/methods to create log entries.
2) Run Chainsaw on my local PC in order to look at the log.  I'd like
to use the "tail" feature to monitor events as they are being
generated.

What kind of connectivity is used by Chainsaw to montitor a remote
file?  Just HTTP?  Do I need to NFS?

I tried creating a receiver.  It asks for a file name -- but not a UNC
or server name/directory location.  How do I specify this information?

TIA!,
-Craig

> The the new version of log4j's Chainsaw, configured to use a
> LogFilePatternReceiver, will do exactly this.
[quoted text clipped - 33 lines]
> >
> > Thanks.
scott - 29 Jun 2004 06:04 GMT
Hi Craig,

Receivers are new to log4j1.3, which isn't released yet (will be in
alpha soon), but we're using them with Chainsaw).  We'll work on
improving the docs.

LogFilePatternReceiver expects a path to look like this:
c:/somepath/somefile.

We're working on adding the ability to process files accessible via
VFS, but that'll be a while before it's available.  Right now
LogFilePatternReceiver expects a file path, so you'll probably need to
use NFS for now.

By the way, Chainsaw has built in support for ORO regular expressions
(define a regexp using the LIKE operator).  You can also combine
expressions using parentheses and logical operators (&&, ||, !).  Just
make sure to put spaces around all operators, operands and
parentheses.

If you have fields in your log file that don't match to log4j's
built-in fields, specify the field(s) in the logFormat as PROP(MYKEY)
for example, and the field will end up in as the 'MYKEY' column in
Chainsaw, which can then be used to filter expressions.

Note that you can specify a filter expression, which can be used to
pre-filter events from being processed by Chainsaw.  Leave this param
out if you want to load the entire file.

If possible, specify a logger in the logFormat - it can be used in the
logger tree panel to quickly access events without having to build an
expression rule.

The receiver may not process events if a logger field isn't specified
(I can't recall).

Here's a sample log4j.xml file that can be used as a Chainsaw
configuration file that will process your file's events on Chainsaw's
startup (specify this xml file here: view menu, show application-wide
preferences, automatic configuration.

log4j.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"
debug="true">
  <plugin name="LogFileReceiver"
class="org.apache.log4j.varia.LogFilePatternReceiver">
     <param name="timestampFormat" value="yyyy-MM-d HH:mm:ss,SSS"/>
     <param name="logFormat" value="RELATIVETIME [THREAD] LEVEL
LOGGER * - MESSAGE"/>
     <param name="fileName" value="c:/test/A4.log"/>
     <param name="filterExpression" value="LEVEL > WARN"/>
     <param name="tailing" value="true"/>
  </plugin>
  <root>
     <level value="debug"/>
  </root>
</log4j:configuration>

I hope this info helps.  If you have further questions, let me know.

Scott

sdeboy@apache.org

> Ok Scott/All, you got me interested.  I downloaded Chainsaw, but the
> instructions on creating receivers is almost non-existant.  Here's
[quoted text clipped - 53 lines]
> > >
> > > Thanks.
Tina - 09 Nov 2005 15:40 GMT
Hi, Scott, I tried to set the Automatic Configuration URL to
file:///C:/chanisaw/log4j.xml
and the log4j.xml is the one you provided here. But when I run chainsaw, the
Receiver's window shows no receivers defined. I right click on the receiver
and choose new receiver, it  only gives you option of
MulticastReceiver
SocketHubReceiver
SocketReceiver
UDPReceiver
XMLSpcketReceiver

and no LogFileReceiver displaied. I am using Chainsaw 2.0alpha.

Any idea?

Thanks!

Tina

>Hi Craig,
>
[quoted text clipped - 68 lines]
>> > >
>> > > Thanks.
Thomas Weidenfeller - 28 Jun 2004 09:39 GMT
> Are there tools to parse application log files (typically stdout & stderr,
> but also Log4J output) in a smart way ?
>
> Basically, I'd like to parse them and compute stats like when, how many
> times and how often exceptions of a class have been raised.

Perl had been invented to do things like this. But also the classic Unix
tools like awk are still not bad at it. Construct regular expressions to
match the format of the lines you are interested in, and use a hash
(Perl) or associative array to count per event, class, or whatever you want.

If you need some graphics, consider to generate simple tables from the
accumulated data, and run gnuplot on them.

/Thomas


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.