Security

adding custom metadata to large indexed file

dotan_patrich77
New Member

I'm indexing files collected from the network using a sinkhole and need to add to the events in the file additional custom fields (i.e. network name, collection time etc.)
I thought about using the added metadata to file as described here - http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Assignmetadatatoeventsdynamically
however, the documentation says the metadata line will be applied to events following the file content where the *** SPLUNK *** metadata line is placed.
Since the indexed file can be very large (1GB), I cannot place the splunk metadata line at the beggining of the file and thought about placing it at the end of the fail and than use tail to index the file backwards. Will it work and the metadata line will be applied to the entire file?

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

No you can't. Splunk never reads a file backwards. If it starts at the end, it simply skips the beginning and reads forward.

If you're trying to do this, I would suggest you simply create a new batch/sinkhole input in inputs.conf, pointing at a different location, or use the the oneshot command to simply index each file on demand. Note also that you can't add arbitrary index-time fields to your data using the ***SPLUNK*** method anyway (only specific Splunk keys like host, source, sourcetype, index), and would need to use additional TRANSFORMS to make it work. You might have similar problems using any other method, however.

Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...