Getting Data In

If an indexer receives logs from a heavy forwarder is it able to forward a subset of data to another Indexer?

gcusello
SplunkTrust
SplunkTrust

Hi at all,
I have some Heavy Forwarders that receive data from some Universal Forwarders and take syslogs from some appliances.
HFs take these logs and forward all of them to an Indexer (Indexer_A) and all the flow runs correctly.

My problem is that I have to forward a part of these logs to another Splunk (Indexer_B) (I know that these logs are indexed two times!):
I configured Indexer_A as described in https://docs.splunk.com/Documentation/Splunk/8.0.1/Forwarding/Routeandfilterdatad but I receive in Indexer_B all the data of Indexer_A, also Splunk internal logs.
These are my configurations on Indexer_A

outputs.conf

[tcpout]
indexAndForward = 1

[tcpout:Indexer_B]
disabled = 0
server = 1.1.1.1:9997

props.conf

[WinEventLog:Security]
TRANSFORMS-routing=fw_win

transforms.conf

[fw_win]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=Indexer_B

I'd like to forward only WinEventLog:Security logs but I receinve also other logs, also Splunk internal logs.

My question is: if Indexer_A receives logs from an Heavy Forwarder is it able to forward a subset of data to another Indexer or must I do this on HFs?
If it's possible what I'm doing wrong?

Ciao and thank you.
Giuseppe

0 Karma
1 Solution

jarizeloyola
Path Finder

Based from my analysis, it's not working because the data is already cooked when it reaches indexer_A, so those props/transforms do nothing. Those props/transforms aren't executed on indexer_a at all, because the HF have parsed the data already, sent it to indexer_a as "cooked data", and therefore indexer_a will just put them to disk and skip it's local props/transforms.

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

Wanted to confirm jarizeloyola answer, you need to do this at the source enterprise server or not pass the data through a HF at all!

You can re-cook data but it is unsupported and would not be a good idea on the indexing tier...

You might be able to use CLONE_SOURCETYPE at the HF and then route it with changes refer to:
https://answers.splunk.com/answers/556300/how-can-i-use-clone-sourcetype-to-send-a-cloned-mo.html

jarizeloyola
Path Finder

Based from my analysis, it's not working because the data is already cooked when it reaches indexer_A, so those props/transforms do nothing. Those props/transforms aren't executed on indexer_a at all, because the HF have parsed the data already, sent it to indexer_a as "cooked data", and therefore indexer_a will just put them to disk and skip it's local props/transforms.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jarizeloyola,
thank you very much, for you help!
it's a liitation!.

Ciao and next time.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...