topic Re: Extract index from filename in inputs.conf in Getting Data In

Extract index from filename in inputs.conf

lindsaylandry — Wed, 27 Dec 2017 21:58:02 GMT

We have a splunkforwarder DaemonSet in Kubernetes, which is forwarding node logs to our splunk server.

We want to take the STDOUT logs from each container, located in /var/log/containers/*.log, and index by the namespace specified in the filename. Is there a way to do this?

Filenames look as follows:

/var/log/containers/<pod-name>_<namespace>_<some-hash>.log

We'd like to set the index in inputs.conf by extracting the middle namespace from these files. I know there is a host_regex that will dynamically set the host, but I haven't found an equivalent for index.

Re: Extract index from filename in inputs.conf

gwalford — Wed, 27 Dec 2017 22:32:34 GMT

You probably want the solution found here:

https://answers.splunk.com/answers/145389/routing-data-to-specific-index-based-on-filename.html

Re: Extract index from filename in inputs.conf

micahkemp — Wed, 27 Dec 2017 22:45:45 GMT

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

Re: Extract index from filename in inputs.conf

lindsaylandry — Fri, 29 Dec 2017 15:37:47 GMT

this is good for the server when it gets the data, but is there a way to change the index on the universal forwarder side?

Re: Extract index from filename in inputs.conf

micahkemp — Fri, 29 Dec 2017 15:52:42 GMT

There is not. You would need to place this configuration on the first heavy forwarder or indexer that sees the data.