Security

adding custom metadata to large indexed file

dotan_patrich77
New Member

I'm indexing files collected from the network using a sinkhole and need to add to the events in the file additional custom fields (i.e. network name, collection time etc.)
I thought about using the added metadata to file as described here - http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Assignmetadatatoeventsdynamically
however, the documentation says the metadata line will be applied to events following the file content where the *** SPLUNK *** metadata line is placed.
Since the indexed file can be very large (1GB), I cannot place the splunk metadata line at the beggining of the file and thought about placing it at the end of the fail and than use tail to index the file backwards. Will it work and the metadata line will be applied to the entire file?

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

No you can't. Splunk never reads a file backwards. If it starts at the end, it simply skips the beginning and reads forward.

If you're trying to do this, I would suggest you simply create a new batch/sinkhole input in inputs.conf, pointing at a different location, or use the the oneshot command to simply index each file on demand. Note also that you can't add arbitrary index-time fields to your data using the ***SPLUNK*** method anyway (only specific Splunk keys like host, source, sourcetype, index), and would need to use additional TRANSFORMS to make it work. You might have similar problems using any other method, however.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...