Knowledge Management

Trying to get two sourcetypes moved from one index to another at indexing time

nls7010
Path Finder

One of my clients has an app in a container. I am unable to modify the index it goes to. I would like to put two of the sourcetypes that come from that host in a different index (by default it puts it in the main index). This is what I have so far, but wanted to know if this configuration would work:
props.conf

[mysourcetype1]
TRANSFORMS-index1 = overrideindex

[mysourcetype2]
TRANSFORMS-index1 = overrideindex

transforms.conf

[overrideindex1]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_new_index

[overrideindex2]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_other_new_index

0 Karma

nls7010
Path Finder

For the application we created, the sourcetypes are going to the correct index. Unfortunately, the "default" container install on Diamonti, which we can't change, sends logs to the main index. So, because of that setup, I need to change on the indexers before the actual indexing is done. Since I can't control it with the app we push out to the containers, I need to have it happen once the indexers begin to receive the "default" sourcetypes from the containers.

0 Karma

nls7010
Path Finder

Unfortunately I don't have access to the configuration on the diamonti containers so I can't put it in the inputs.conf. I have to catch these AFTER they are sent to the index (main). We have an inputs.conf on the container, but these are logs we couldn't control with inputs.conf, they are part of an automatic feed to the main index.

0 Karma

koshyk
Super Champion

But how are you pushing these inputs.conf? I meant via deployment server or via any orchestration mechanism like ansible/puppet?

0 Karma

koshyk
Super Champion

I think you have put the exact same name in both transforms stanza

[mysourcetype1]
TRANSFORMS-index1 = overrideindex1

[mysourcetype2]
TRANSFORMS-index2 = overrideindex2

Also, why are you trying to do at props/transforms level? This is more intensive right as every event needs to be transformed. It may be easy to assign a new sourcetype & different index at inputs.conf level itself in the client system

eg . in inputs.conf

[monitor:///var/log/file1.log]
index = my_new_index
sourcetype = mysourcetype1

[monitor:///var/log/file2.log]
index = my_other_new_index
sourcetype = mysourcetype2
0 Karma

nls7010
Path Finder

Forgot to put the 1 and 2 in the props.conf above. So I have them as 1 & 2 props and 1 & 2 transforms

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...