Getting Data In

How to route an index based on a search-time extracted field?

mspetrovic
Engager

I wish to use a search-time extracted field as the basis for routing to a specific index.

In my props.conf file I have a search-time extraction of fields. These fields are taken from the monitored log file name, which is a name that follows a known pattern but whose exact name cannot be predicted. Here is a typical log file name:

/var/log/containers/coredns-845cddccfc-prm27_kube-system_coredns-4a676a5795e68f4f2bacaed884bc3811de019083801b803ad67974b4ac3c221d.log

The set of log files are of a common sourcetype=kubernetes, evidenced in the LWF inputs.conf

[monitor:///var/log/containers/*.log]
sourcetype = kubernetes
multiline_event_extra_waittime = true

In props.conf I have

[kubernetes]
EXTRACT-sourcefields = /var/log/containers/(?<pod>.*)_(?<namespace>.*)_(?<containerName>.*)-(?<containerID>.*)\.log in source

To route to an index of interest, I add this to my props.conf

TRANSFORMS-theindex = theindex

In transforms.conf I add the associated stanza for this index, with the intent of routing to myindex when containerName=aContainerName. I note that the index myindex does in fact exist.

[theindex]
SOURCE_KEY = field:containerName
REGEX = aContainerName
DEST_KEY = _MetaData:Index
FORMAT = myindex

However, I find that the myindex never accrues events.

I suspect I am not allowed to use search-time extracted fields in transforms. If that is true, and my desire to route based on an extracted field, what approach should I take?

0 Karma
1 Solution

FrankVl
Ultra Champion

Search time extractions are exactly that: search time. So that information is not available at index time.

But since you are using the source field, which of course is available at index time, you should simply be able to re-use a very similar expression as used for the search time extraction, to write your transforms. Just use the source field as the source_key.

Alternatively, since you're using the source field, you might even be able to write this much simpler:

props.conf

[source::/var/log/containers/*_*_aContainerName-*]
TRANSFORMS-theindex = theindex

transforms.conf

[theindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = myindex

View solution in original post

FrankVl
Ultra Champion

Search time extractions are exactly that: search time. So that information is not available at index time.

But since you are using the source field, which of course is available at index time, you should simply be able to re-use a very similar expression as used for the search time extraction, to write your transforms. Just use the source field as the source_key.

Alternatively, since you're using the source field, you might even be able to write this much simpler:

props.conf

[source::/var/log/containers/*_*_aContainerName-*]
TRANSFORMS-theindex = theindex

transforms.conf

[theindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = myindex

mspetrovic
Engager

Thank you. Using your guidance, this stanza in transforms.conf meets my need:

[theindex]
SOURCE_KEY = MetaData:Source
# source sample: /var/log/containers/foo_default_containerName-f443343eb91ab92b1ed477bdc8b313727171214c0d7f3e1f13222ea786aaec5f.log
REGEX = /var/log/containers/?.*_?.*_containerName-.*\.log
DEST_KEY = _MetaData:Index
FORMAT = myindex
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...