Knowledge Management

Trying to get two sourcetypes moved from one index to another at indexing time

nls7010
Path Finder

One of my clients has an app in a container. I am unable to modify the index it goes to. I would like to put two of the sourcetypes that come from that host in a different index (by default it puts it in the main index). This is what I have so far, but wanted to know if this configuration would work:
props.conf

[mysourcetype1]
TRANSFORMS-index1 = overrideindex

[mysourcetype2]
TRANSFORMS-index1 = overrideindex

transforms.conf

[overrideindex1]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_new_index

[overrideindex2]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_other_new_index

0 Karma

nls7010
Path Finder

For the application we created, the sourcetypes are going to the correct index. Unfortunately, the "default" container install on Diamonti, which we can't change, sends logs to the main index. So, because of that setup, I need to change on the indexers before the actual indexing is done. Since I can't control it with the app we push out to the containers, I need to have it happen once the indexers begin to receive the "default" sourcetypes from the containers.

0 Karma

nls7010
Path Finder

Unfortunately I don't have access to the configuration on the diamonti containers so I can't put it in the inputs.conf. I have to catch these AFTER they are sent to the index (main). We have an inputs.conf on the container, but these are logs we couldn't control with inputs.conf, they are part of an automatic feed to the main index.

0 Karma

koshyk
Super Champion

But how are you pushing these inputs.conf? I meant via deployment server or via any orchestration mechanism like ansible/puppet?

0 Karma

koshyk
Super Champion

I think you have put the exact same name in both transforms stanza

[mysourcetype1]
TRANSFORMS-index1 = overrideindex1

[mysourcetype2]
TRANSFORMS-index2 = overrideindex2

Also, why are you trying to do at props/transforms level? This is more intensive right as every event needs to be transformed. It may be easy to assign a new sourcetype & different index at inputs.conf level itself in the client system

eg . in inputs.conf

[monitor:///var/log/file1.log]
index = my_new_index
sourcetype = mysourcetype1

[monitor:///var/log/file2.log]
index = my_other_new_index
sourcetype = mysourcetype2
0 Karma

nls7010
Path Finder

Forgot to put the 1 and 2 in the props.conf above. So I have them as 1 & 2 props and 1 & 2 transforms

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...