Knowledge Management

Trying to get two sourcetypes moved from one index to another at indexing time

nls7010
Path Finder

One of my clients has an app in a container. I am unable to modify the index it goes to. I would like to put two of the sourcetypes that come from that host in a different index (by default it puts it in the main index). This is what I have so far, but wanted to know if this configuration would work:
props.conf

[mysourcetype1]
TRANSFORMS-index1 = overrideindex

[mysourcetype2]
TRANSFORMS-index1 = overrideindex

transforms.conf

[overrideindex1]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_new_index

[overrideindex2]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = my_other_new_index

0 Karma

nls7010
Path Finder

For the application we created, the sourcetypes are going to the correct index. Unfortunately, the "default" container install on Diamonti, which we can't change, sends logs to the main index. So, because of that setup, I need to change on the indexers before the actual indexing is done. Since I can't control it with the app we push out to the containers, I need to have it happen once the indexers begin to receive the "default" sourcetypes from the containers.

0 Karma

nls7010
Path Finder

Unfortunately I don't have access to the configuration on the diamonti containers so I can't put it in the inputs.conf. I have to catch these AFTER they are sent to the index (main). We have an inputs.conf on the container, but these are logs we couldn't control with inputs.conf, they are part of an automatic feed to the main index.

0 Karma

koshyk
Super Champion

But how are you pushing these inputs.conf? I meant via deployment server or via any orchestration mechanism like ansible/puppet?

0 Karma

koshyk
Super Champion

I think you have put the exact same name in both transforms stanza

[mysourcetype1]
TRANSFORMS-index1 = overrideindex1

[mysourcetype2]
TRANSFORMS-index2 = overrideindex2

Also, why are you trying to do at props/transforms level? This is more intensive right as every event needs to be transformed. It may be easy to assign a new sourcetype & different index at inputs.conf level itself in the client system

eg . in inputs.conf

[monitor:///var/log/file1.log]
index = my_new_index
sourcetype = mysourcetype1

[monitor:///var/log/file2.log]
index = my_other_new_index
sourcetype = mysourcetype2
0 Karma

nls7010
Path Finder

Forgot to put the 1 and 2 in the props.conf above. So I have them as 1 & 2 props and 1 & 2 transforms

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...