Knowledge Management

Trying to get two sourcetypes moved from one index to another at indexing time

nls7010
Path Finder

One of my clients has an app in a container. I am unable to modify the index it goes to. I would like to put two of the sourcetypes that come from that host in a different index (by default it puts it in the main index). This is what I have so far, but wanted to know if this configuration would work:
props.conf

[mysourcetype1]
TRANSFORMS-index1 = overrideindex

[mysourcetype2]
TRANSFORMS-index1 = overrideindex

transforms.conf

[overrideindex1]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_new_index

[overrideindex2]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_other_new_index

0 Karma

nls7010
Path Finder

For the application we created, the sourcetypes are going to the correct index. Unfortunately, the "default" container install on Diamonti, which we can't change, sends logs to the main index. So, because of that setup, I need to change on the indexers before the actual indexing is done. Since I can't control it with the app we push out to the containers, I need to have it happen once the indexers begin to receive the "default" sourcetypes from the containers.

0 Karma

nls7010
Path Finder

Unfortunately I don't have access to the configuration on the diamonti containers so I can't put it in the inputs.conf. I have to catch these AFTER they are sent to the index (main). We have an inputs.conf on the container, but these are logs we couldn't control with inputs.conf, they are part of an automatic feed to the main index.

0 Karma

koshyk
Super Champion

But how are you pushing these inputs.conf? I meant via deployment server or via any orchestration mechanism like ansible/puppet?

0 Karma

koshyk
Super Champion

I think you have put the exact same name in both transforms stanza

[mysourcetype1]
TRANSFORMS-index1 = overrideindex1

[mysourcetype2]
TRANSFORMS-index2 = overrideindex2

Also, why are you trying to do at props/transforms level? This is more intensive right as every event needs to be transformed. It may be easy to assign a new sourcetype & different index at inputs.conf level itself in the client system

eg . in inputs.conf

[monitor:///var/log/file1.log]
index = my_new_index
sourcetype = mysourcetype1

[monitor:///var/log/file2.log]
index = my_other_new_index
sourcetype = mysourcetype2
0 Karma

nls7010
Path Finder

Forgot to put the 1 and 2 in the props.conf above. So I have them as 1 & 2 props and 1 & 2 transforms

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...