 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Java Forum / General / March 2008Monitoring directory for new files on Solaris
Hello! An application we need to write has to monitor a directory and process any files appearing in there -- as quickly as possible. The usual approach to this task is to rescan the directory every once in a while, which is horrible -- it loads the system keeping the program in RAM and still has latency measured in seconds... How can I ask the kernel to notify my program, when the directory is updated in any way -- via something like poll(), or select(), or http://developers.sun.com/solaris/articles/event_completion.html ? Is there a Java way of doing this already (without periodic polling), or will I need to make my own JNI class? Thanks for any pointers! -mi > Hello! > [quoted text clipped - 18 lines] > Is there a Java way of doing this already (without periodic polling), or > will I need to make my own JNI class? from the docs for JNotify: "JNotify works on both Windows (Windows 2000, XP, Vista) and Linux with INotify support (Kernel 2.6.14 and above)." This probably won't help you much if you're on Solaris. java-fam may give you some ideas. Mark Mikhail Teterin schrieb: > Hello! > [quoted text clipped - 4 lines] > while, which is horrible -- it loads the system keeping the program in RAM > and still has latency measured in seconds... what is a rescan in your sense? to scan an mostly empty dir, why should that be slow, dont you empty dir after processing? Which language your app use, pure java? first idea (not java): install gnu find, scan all 5 seconds for file newer then 5 sec., but take care (read last sentence). second idea: in filesystem terms: just look for changes in the directory, like (again just system commands): "cat /dir > /tmp/dirread1 ; sleep 5 ; cat /dir > /tmp/dirread2 ; diff dirread1 dirread2" (sorry, at the moment I can not proof what cat does on dirs, maybe dd does the trick). are there any routines in C or java for just reading the dir themself, not the inodes in it? In my opinion: the more difficult part in your software are the file locking during processing the files and to ashure not to process files which are still in access of other processes, e.g. which still not complete or just not closed by the writing process. W. Wolfgang schrieb: > second idea: in filesystem terms: just look for changes in the > directory, I forgot (or in german: den Wald vor lauter Baeumen nicht sehen): as mentioned in another post, unix has also for directorys as for every inode the atime/mtime/ctime. I think to monitor just the mtime of the dir give you the trigger to scan for new files, if you dont have subdirs. W. > Mikhail Teterin schrieb: > [quoted text clipped - 13 lines] > first idea (not java): install gnu find, scan all 5 seconds for file > newer then 5 sec., but take care (read last sentence). GNU find has option -cmin, but I could not find option for second. Victor > GNU find has option -cmin, but I could not find option for second. Remember what the most recent file was that you processed, and use "-cnewer" instead. If you have to remove the files as you process them, you can create a marker file of your own solely for this purpose. /gordon -- >> GNU find has option -cmin, but I could not find option for second. > > Remember what the most recent file was that you processed, and use > "-cnewer" instead. If you have to remove the files as you process > them, you can create a marker file of your own solely for this > purpose. On the other hand, there is no need to use find when your existing application already does exactly this kind of scanning. Find doesn't offer any shortcuts here. /gordon -- > what is a rescan in your sense? to scan an mostly empty dir, why should > that be slow, dont you empty dir after processing? Which language your > app use, pure java? > first idea (not java): install gnu find, scan all 5 seconds for file > newer then 5 sec., but take care (read last sentence). The actual rescan is not difficult to implement -- as others (and yourself) have pointed out, one just needs to check the directory's mtime. However, the problems common with all /periodic/ algorithms is that they are inherently inefficient. You want to be event driven: getting woken up, when stuff happens, instead of -- as bored children do to their parents when riding in a car -- continuously asking the kernel: "Are we there yet? No? How about now? Are we there yet?" For example, you suggested 5 seconds, so let's go with that... Polling means, your program must always reside in memory -- the OS can not really page it out, because the program needs to run every 5 seconds. On the other hand, your average latency will still be 2.5 seconds (and up to 5), which may well be too long for your application... Moving the 5 seconds figure into any direction eases one of these two negatives while exacerbating the other... Better OSes have a way for programs to request notification from the OS -- the kernel knows, when it changes things, so why can't it tell the process, if asked nicely? Unfortunately, Solaris has not joined this club yet, according to the link kindly posted by Ian Collins in this thread: http://blogs.sun.com/praks/entry/file_events_notification For now, however, no good solution appears to exist and I will have to use polling... Yours, -mi In comp.unix.solaris Mikhail Teterin <usenet+mill@aldan.algebra.com> wrote: > http://blogs.sun.com/praks/entry/file_events_notification > > For now, however, no good solution appears to exist and I will have to use > polling... It's not the best solution, but you could use 'dtrace' on Solaris 10. It would not require polling, but recieves probes as events. That said, I don't think you can guarantee its correctness. You'd want to fall back on polling (at a longer frequency) as a safety check.  SignatureDarren Dunham ddunham@taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. > > Hello! > [quoted text clipped - 18 lines] > > -mi I seem to remember that on Unix the modification time of the directory normally changes when there is any change to its content. This may be a bit quicker than listing the content. While repeated listing the directory seems ugly, it isn't all that slow or inefficient (even from Java) as everything is cached. This is especially true if processed files are removed, thus leaving the directory empty. Warning, the directory modification time does not change on Windows, so don't try this approach there. Mark Thornton In comp.unix.solaris Mikhail Teterin <usenet+mill@aldan.algebra.com> wrote: > An application we need to write has to monitor a directory and process any > files appearing in there -- as quickly as possible. > The usual approach to this task is to rescan the directory every once in a > while, which is horrible -- it loads the system keeping the program in RAM > and still has latency measured in seconds... > How can I ask the kernel to notify my program, when the directory is updated > in any way -- via something like poll(), or select(), or > http://developers.sun.com/solaris/articles/event_completion.html Is there something like kqueue() in Solaris? In FreeBSD, many people find the wait_on utility very useful. See http://freebsd.unixfreunde.de/sources/wait_on-1.1.tar.gz SignatureVictor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ > In comp.unix.solaris Mikhail Teterin <usenet+mill@aldan.algebra.com> wrote: > [quoted text clipped - 13 lines] > In FreeBSD, many people find the wait_on utility very useful. > See http://freebsd.unixfreunde.de/sources/wait_on-1.1.tar.gz There isn't anything in Solaris that does this but with OpenSolaris, you could mod the kernel or maybe create a Dtrace script that "wakes up" when a directory is modified. SignatureDeeDee, don't press that button! DeeDee! NO! Dee... > An application we need to write has to monitor a directory and > process any files appearing in there -- as quickly as possible. > > The usual approach to this task is to rescan the directory every > once in a while, which is horrible -- it loads the system keeping > the program in RAM and still has latency measured in seconds... Three suggestions: - Poll the last modified time of the directory itself at a suitable interval. It will be updated when changes are made to the directory (e.g. files added, removed, renamed). When you see that something has changed, rescan the directory to see what it was. - I believe Solaris has an mechanism for subscribing to file system events. Write a JNI wrapper for that. - Modify the application producing these files, so that it can notify your application. /gordon -- In comp.unix.solaris Gordon Beaton <n.o.t@for.email> wrote: > - I believe Solaris has an mechanism for subscribing to file system > events. Could you please elaborate. SignatureVictor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ > In comp.unix.solaris Gordon Beaton <n.o.t@for.email> wrote: >> - I believe Solaris has an mechanism for subscribing to file system >> events. > > Could you please elaborate. If you are using Solaris Express there's this: http://blogs.sun.com/praks/entry/file_events_notification SignatureIan Collins. In comp.unix.solaris Ian Collins <ian-news@hotmail.com> wrote: > >> - I believe Solaris has an mechanism for subscribing to file system > >> events. > > > > Could you please elaborate. > > > If you are using Solaris Express there's this: > http://blogs.sun.com/praks/entry/file_events_notification This is not official, I presume. Not supported by Sun, is it? SignatureVictor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ >In comp.unix.solaris Ian Collins <ian-news@hotmail.com> wrote: >> >> - I believe Solaris has an mechanism for subscribing to file system [quoted text clipped - 3 lines] >> > >> If you are using Solaris Express there's this: >> http://blogs.sun.com/praks/entry/file_events_notification >This is not official, I presume. Not supported by Sun, is it? What do you mean by "not official"? It's not in Solaris 10 or earlier if that is what you mean. Casper SignatureExpressed in this posting are my opinions. They are in no way related to opinions held by my employer, Sun Microsystems. Statements on Sun products included here are not gospel and may be fiction rather than truth. In comp.unix.solaris Casper H.S. Dik <Casper.Dik@sun.com> wrote: > >> >> - I believe Solaris has an mechanism for subscribing to file system > >> >> events. > >> > > >> > Could you please elaborate. > >> > > >> If you are using Solaris Express there's this: > >> http://blogs.sun.com/praks/entry/file_events_notification > >This is not official, I presume. Not supported by Sun, is it? > What do you mean by "not official"? It's not in Solaris 10 or > earlier if that is what you mean. I mean it's not from Sun but from some volunteer contributor, is it? SignatureVictor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ On Mar 14, 1:38 pm, Victor Sudakov <v...@mpeks.no-spam-here.tomsk.su> wrote: > In comp.unix.solaris Casper H.S. Dik <Casper....@sun.com> wrote: > [quoted text clipped - 10 lines] > > I mean it's not from Sun but from some volunteer contributor, is it? Well, as he has his blog on sun.com he is more than a volunteer... ;-) > If you are using Solaris Express there's this: > > http://blogs.sun.com/praks/entry/file_events_notification Yes, this would be what I'm looking for, except Sun is not shipping it -- it will not be in the production hardware for years :( Damn it, *BSD had kqueue/kevent for years already, but I'm stuck on Solaris... -mi Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.