topic Re: job based log file processing in Getting Data In

job based log file processing

levinro — Tue, 26 Nov 2013 21:57:11 GMT

Newbie here - Just evaluating Splunk.

I set up my source to watch a directory and my source type filtering by file name pattern. This is all great, it parsed out errors & messages nicely with no effort.

Now I want to take it a step further. When each file arrives and is closed (the discrete job finished), I want to evaluate things like run time (time between last & first time stamp in the log file) and record counts (a string pulled from a specific message in the log file). Then I want to create trending reports on these values on a daily/weekly/monthly basis.

Re: job based log file processing

lukejadamec — Wed, 27 Nov 2013 01:09:08 GMT

Individual log files are generally considered to be a Source, not sure if you can do a transaction on a Source.

Re: job based log file processing

jtacy — Wed, 27 Nov 2013 04:10:38 GMT

Using metadata is probably the fastest way to get the start/end timestamps for each source. You can then get the difference in seconds pretty easily. You'll need to have some kind of search that returns the record count for each file and that might involve using rex to pull that field out. That field extraction should happen in the subsearch part of the example below:

| metadata type=sources index=yourindex | eval duration=lastTime-firstTime | join source [search index=yourindex recordcount=* | top 1 recordcount by source] | table source, duration, recordcount

If you're going to have a lot of events, optimizing that subsearch will be crucial when you start getting into those weekly/monthly groupings. There will be some stuff to add to turn this into a chart, but I'd probably start with the field extraction and getting the duration and record count into a table and go from there. Good luck!