Security

adding custom metadata to large indexed file

dotan_patrich77
New Member

I'm indexing files collected from the network using a sinkhole and need to add to the events in the file additional custom fields (i.e. network name, collection time etc.)
I thought about using the added metadata to file as described here - http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Assignmetadatatoeventsdynamically
however, the documentation says the metadata line will be applied to events following the file content where the *** SPLUNK *** metadata line is placed.
Since the indexed file can be very large (1GB), I cannot place the splunk metadata line at the beggining of the file and thought about placing it at the end of the fail and than use tail to index the file backwards. Will it work and the metadata line will be applied to the entire file?

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

No you can't. Splunk never reads a file backwards. If it starts at the end, it simply skips the beginning and reads forward.

If you're trying to do this, I would suggest you simply create a new batch/sinkhole input in inputs.conf, pointing at a different location, or use the the oneshot command to simply index each file on demand. Note also that you can't add arbitrary index-time fields to your data using the ***SPLUNK*** method anyway (only specific Splunk keys like host, source, sourcetype, index), and would need to use additional TRANSFORMS to make it work. You might have similar problems using any other method, however.

Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...