job based log file processing

[levinro]

Newbie here - Just evaluating Splunk.

I set up my source to watch a directory and my source type filtering by file name pattern. This is all great, it parsed out errors & messages nicely with no effort.

Now I want to take it a step further. When each file arrives and is closed (the discrete job finished), I want to evaluate things like run time (time between last & first time stamp in the log file) and record counts (a string pulled from a specific message in the log file). Then I want to create trending reports on these values on a daily/weekly/monthly basis.

[jtacy]

Using metadata is probably the fastest way to get the start/end timestamps for each source. You can then get the difference in seconds pretty easily. You'll need to have some kind of search that returns the record count for each file and that might involve using rex to pull that field out. That field extraction should happen in the subsearch part of the example below:

| metadata type=sources index=yourindex | eval duration=lastTime-firstTime | join source [search index=yourindex recordcount=* | top 1 recordcount by source] | table source, duration, recordcount

If you're going to have a lot of events, optimizing that subsearch will be crucial when you start getting into those weekly/monthly groupings. There will be some stuff to add to turn this into a chart, but I'd probably start with the field extraction and getting the duration and record count into a table and go from there. Good luck!

[lukejadamec]

Individual log files are generally considered to be a Source, not sure if you can do a transaction on a Source.