Extract index from filename in inputs.conf

[lindsaylandry]

We have a splunkforwarder DaemonSet in Kubernetes, which is forwarding node logs to our splunk server.

We want to take the STDOUT logs from each container, located in /var/log/containers/*.log, and index by the namespace specified in the filename. Is there a way to do this?

Filenames look as follows:

/var/log/containers/<pod-name>_<namespace>_<some-hash>.log

We'd like to set the index in inputs.conf by extracting the middle namespace from these files. I know there is a host_regex that will dynamically set the host, but I haven't found an equivalent for index.

[micahkemp · Accepted solution]

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

[lindsaylandry]

this is good for the server when it gets the data, but is there a way to change the index on the universal forwarder side?

[micahkemp]

There is not. You would need to place this configuration on the first heavy forwarder or indexer that sees the data.

[gwalford]

You probably want the solution found here:

https://answers.splunk.com/answers/145389/routing-data-to-specific-index-based-on-filename.html