Getting Data In

Extract index from filename in inputs.conf

lindsaylandry
Engager

We have a splunkforwarder DaemonSet in Kubernetes, which is forwarding node logs to our splunk server.

We want to take the STDOUT logs from each container, located in /var/log/containers/*.log, and index by the namespace specified in the filename. Is there a way to do this?

Filenames look as follows:

/var/log/containers/<pod-name>_<namespace>_<some-hash>.log

We'd like to set the index in inputs.conf by extracting the middle namespace from these files. I know there is a host_regex that will dynamically set the host, but I haven't found an equivalent for index.

0 Karma
1 Solution

micahkemp
Champion

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

View solution in original post

micahkemp
Champion

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

lindsaylandry
Engager

this is good for the server when it gets the data, but is there a way to change the index on the universal forwarder side?

0 Karma

micahkemp
Champion

There is not. You would need to place this configuration on the first heavy forwarder or indexer that sees the data.

0 Karma

gwalford
Path Finder
0 Karma
Get Updates on the Splunk Community!

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...

From Alert to Resolution: How Splunk Observability Helps SREs Navigate Critical ...

It's 3:17 AM, and your phone buzzes with an urgent alert. Wire transfer processing times have spiked, and ...