Security

adding custom metadata to large indexed file

dotan_patrich77
New Member

I'm indexing files collected from the network using a sinkhole and need to add to the events in the file additional custom fields (i.e. network name, collection time etc.)
I thought about using the added metadata to file as described here - http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Assignmetadatatoeventsdynamically
however, the documentation says the metadata line will be applied to events following the file content where the *** SPLUNK *** metadata line is placed.
Since the indexed file can be very large (1GB), I cannot place the splunk metadata line at the beggining of the file and thought about placing it at the end of the fail and than use tail to index the file backwards. Will it work and the metadata line will be applied to the entire file?

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

No you can't. Splunk never reads a file backwards. If it starts at the end, it simply skips the beginning and reads forward.

If you're trying to do this, I would suggest you simply create a new batch/sinkhole input in inputs.conf, pointing at a different location, or use the the oneshot command to simply index each file on demand. Note also that you can't add arbitrary index-time fields to your data using the ***SPLUNK*** method anyway (only specific Splunk keys like host, source, sourcetype, index), and would need to use additional TRANSFORMS to make it work. You might have similar problems using any other method, however.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...