topic Re: Why is data received from a remote Splunk instance not being collected in the specified index? in Getting Data In

Why is data received from a remote Splunk instance not being collected in the specified index?

adamblock2 — Tue, 29 Sep 2020 07:11:45 GMT

Our Splunk instance is currently receiving data from a remote Splunk instance. The remote indexer is sending data (many hosts with many different sourcetypes) to our indexers over TCP port 9998. We are interested in forcing this data to be collected in a custom index.

I have confirmed that we are receiving data from the remote Splunk on port 9998, however, it is not being collected in the desired index. The following are the inputs.conf, props.conf, and transforms.conf which I currently have in place:

inputs.conf

[splunktcp://:9998]

index=CustomIndex

props.conf

[source::tcp:9998]

TRANSFORMS-force_index = setIndexMeta

transforms.conf

[setIndexMeta]
DEFAULT_VALUE = unknown
REGEX = (.)
DEST_KEY = _MetaData:Index

FORMAT = CustomIndex

I would appreciate assistance with this.

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

woodcock — Fri, 11 Sep 2015 03:46:11 GMT

Forget props.conf and transforms.conf and just do this inside $SPLUNK_HOME/etc/apps/MyApp/default/inputs.conf:

[splunktcp://:9998]
index=CustomIndex
sourcetype=MySourceType

You also need to make sure you check/add this inside $SPLUNK_HOME/etc/system/local/default-mode.conf (it defaults to disabled):

[pipeline:tcp]
disabled = false

Then you need to bounce all Splunk instances on the servers that get these files.

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

MuS — Fri, 11 Sep 2015 03:52:34 GMT

In addition: The lines in default-mode.conf are only needed if this will be done on a forwarder.

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

cramasta — Fri, 11 Sep 2015 04:35:54 GMT

I thought this was because the intermediate indexer sending cooked data to the final indexer? If so I was thinking that using the route settings as described in this answer would make sure the data goes though the parsing queues again. Does setting you mention for default-mode.conf do something similar? Thanks..

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

woodcock — Fri, 11 Sep 2015 04:51:24 GMT

What do you mean by "intermediate indexer"? What is your architecture?

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

cramasta — Fri, 11 Sep 2015 05:11:17 GMT

Sounds like from the original poster that they are receiving data from another indexer. Thought the data might be fully cooked by the time it gets to him and not go though the parsing queues to set the the new index.

"The remote indexer is sending data to our indexers over TCP port 9998"

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

jclehmuth — Fri, 11 Sep 2015 12:57:08 GMT

It sounds like the OP has a remote Splunk instance and is forwarding data from that instance to his main instance. He wants to force the data coming from the remote instance into a certain "special" index and not into the main/default/or whatever index the remote instance is putting the data into.

If parsing is already done by something else then the indexer is going to ignore the props and transforms, so I see why you said that they can be forgotten. But he has the index in inputs and it seems that it still isn't working.

Re: Why is data received from a remote Splunk instance not being collected in the specified index?

jclehmuth — Tue, 29 Sep 2020 07:11:58 GMT

"When you forward structured data to an indexer, Splunk Enterprise does not parse this data once it arrives at the indexer, even if you have configured props.conf on that indexer with INDEXED_EXTRACTIONS. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:

parsing
aggregation
typing
The forwarded data must arrive at the indexer already parsed."
http://docs.splunk.com/Documentation/Splunk/6.1.4/Forwarding/Routeandfilterdatad

An expensive work around could be done by adding this to the inputs.conf

[splunktcp://9998] route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue
Found here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible.html
http://answers.splunk.com/answers/5528/forwarding-select-data-in-my-environment.html