Archive

adding custom metadata to large indexed file

New Member

I'm indexing files collected from the network using a sinkhole and need to add to the events in the file additional custom fields (i.e. network name, collection time etc.)
I thought about using the added metadata to file as described here - http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Assignmetadatatoeventsdynamically
however, the documentation says the metadata line will be applied to events following the file content where the *** SPLUNK *** metadata line is placed.
Since the indexed file can be very large (1GB), I cannot place the splunk metadata line at the beggining of the file and thought about placing it at the end of the fail and than use tail to index the file backwards. Will it work and the metadata line will be applied to the entire file?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

No you can't. Splunk never reads a file backwards. If it starts at the end, it simply skips the beginning and reads forward.

If you're trying to do this, I would suggest you simply create a new batch/sinkhole input in inputs.conf, pointing at a different location, or use the the oneshot command to simply index each file on demand. Note also that you can't add arbitrary index-time fields to your data using the ***SPLUNK*** method anyway (only specific Splunk keys like host, source, sourcetype, index), and would need to use additional TRANSFORMS to make it work. You might have similar problems using any other method, however.