topic Re: Routing the data to a different Index via Regex in Splunk Search

Routing the data to a different Index via Regex

shiv1593 — Wed, 30 Sep 2020 04:25:24 GMT

Hello,

We have a source ABC sending us logs and are being stored inside an index called all_logs. From that source, we want to separate the events, which contains the field SiteUrl: https://www.abc.com/site/PathologyPHI and send these logs to a new index called new_logs, while the rest of the logs should still go to the Index all_logs.

This is what I've tried so far, but all the logs still land up in the Index all_logs, instead of being routed.. FYI, props and transforms are kept under $SPLUNK_HOME/etc/system/local, while the inputs.conf is under the TA, which brings the logs's local directory. Had to do this because in props, there is a setting defined for another sourcetype as well. Any suggestions regarding moving them back to TA as well will be appreciated.

In props.conf, I made the following change:

[abc:management:activity]
TRANSFORMS-routing = abc_logs

In Transforms.conf, I did this:

[abc_logs]
REGEX = SiteUrl:.+PathologyPHI
DEST_KEY = _MetaData:Index
FORMAT = new_logs

Here is the inputs.conf for the source. There are other inputs too, but this inputs comes with the events, which we want to route. We do not want to route all the events coming off this input, only the ones matching the regex:

[splunk_ta_abc_management_activity://Auditabc]
content_type = Audit.abc
index = all_logs
sourcetype = abc:management:activity
interval = 120
tenant_name = ABC
disabled = 0
start_by_shell = false

Any help will be highly appreciated.

Thank you

Re: Routing the data to a different Index via Regex

gcusello — Thu, 27 Feb 2020 11:26:47 GMT

Hi @shiv1593,
at first, please share again the regex in transforms.conf using the Code Sample button because it isn't possible to read it and check the regex.

Anyway, where do you have props.conf and trasnforms.conf?
they must be on Indexers or (when present) on Heavy Forwarders.

Ciao.
Giuseppe

Re: Routing the data to a different Index via Regex

shiv1593 — Thu, 27 Feb 2020 11:33:32 GMT

Hi Giuseppe,

Please find the regex below:

REGEX = SiteUrl:.+PathologyPHI

Props and transforms are in the heavy forwarder under /etc/system/local. That's where I'm trying to route the data from.

Thank you

Re: Routing the data to a different Index via Regex

gcusello — Thu, 27 Feb 2020 12:29:37 GMT

Hi @shiv1593,
the regex and your approach seem to be correct.
So please some additional info:

A surely stupid question: did you restarted Heavy Forwarder after update?

Logs are in an Universal Forwarder (read by the inputs.conf in the TA) and they are sent to the HF (where there are props.conf and transforms.conf) and then sent to Indexers, is it correct or there's anything else?

I don't understand what you say: "...setting defined for another sourcetype...".

Ciao.
Giuseppe