topic Re: How to Handle Metadata in File Headers in Splunk Search

How to Handle Metadata in File Headers

chrissale — Mon, 10 Feb 2014 13:18:35 GMT

I am using Splunk to collect data from log files generated by a thick client application. The log files contain metadata in the header relating to the user that logged on. I want to be able to search for events using the metadata in the file header (example below).

username: myUser
hostname: myHost
10/02/2014 13:12:03 INFO User did some stuff
10/02/2014 13:12:41 INFO User did some more stuff
10/02/2014 13:14:26 WARNING User did some stuff they weren't supposed to!

In this example a search for 'username=myUser' would return all three events shown. Is that possible?

Re: How to Handle Metadata in File Headers

yannK — Tue, 11 Feb 2014 00:18:34 GMT

Not easily because

the fields username and hostname will be present in the first events (with a timestamp that will be unpredictable)
Then the other events will be all individuals, and not contain the username and hostname information.

If your log files are unique (unique filename), you could build a lookup to link : the source and the username and hostname
then when searching use the lookup to add those fields to the events of this particular source.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/Search/Useexternalfieldlookups

Otherwise, you should change you method of login, and maybe add the username and hostname in the source.
the extract them at search time with a rex command.

or in the path, example : /path/to/my/file/<host>/<username>/file.log
you can have the host field extracted at index time with host_segment or host_regex, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Setadefaulthostforaninput

Re: How to Handle Metadata in File Headers

yannK — Tue, 11 Feb 2014 00:21:30 GMT

I forgot, a subsearch could work if you have unique file sources.

[search username=myUser | dedup source | table source] myothercondition=condition

the sub search will return the source name of the events that contains the username, and it will become a search condition in the man search.

Re: How to Handle Metadata in File Headers

chrissale — Tue, 11 Feb 2014 11:45:55 GMT

Thanks for the response yannK. I like where you are going with this but unfortunately I can't guarantee that the source name is unique or influence the naming strategy for the file. I am going to see if I can create a custom field at indexing time that uniquely identifies the file (CRC?) and use that for the lookup.

Re: How to Handle Metadata in File Headers

yannK — Tue, 11 Feb 2014 18:03:36 GMT

There is a possibility is to add a meta field at index time per monitor in the inputs.conf, but it will not be dynamic per files, So not really what you want.

The only real solution is to format your events to add the fields to every lines before monitoring, Or create a custom scripted input to replace the monitor, and add then on the fly.
see http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/ScriptedInputsIntro