topic Re: Sending uncooked data from indexer level in Getting Data In

Sending uncooked data from indexer level

gyarici — Tue, 11 Feb 2020 16:38:04 GMT

Hi all,

I am sending data from intermediate forwarder to indexer and during indexing, I would like to send raw "uncooked data" to 3rd party application.
Recently I tried to use CEF app index and forward but , it is working but it is becoming cooked data.

Is there any way to handle this from indexer level?

Thanks

Re: Sending uncooked data from indexer level

13tsavage — Tue, 11 Feb 2020 16:58:22 GMT

I am not 100% sure if indexers can forward data the way you are requesting. This would mean that before indexers can parse and begin indexing the data, you want them to first send the uncooked data to other systems.

Re: Sending uncooked data from indexer level

gcusello — Tue, 11 Feb 2020 17:27:01 GMT

Hi @gyarici,
I had the same problem and I received this answer both from Community and Splunk Support:
if you have an intermediate Heavy Forwarder isn't possible to do this on Indexers.
But you can do the fork at Heavy Forwarder level.

Ciao.
Giuseppe

Re: Sending uncooked data from indexer level

codebuilder — Tue, 11 Feb 2020 17:54:16 GMT

This example should accomplish what you are trying to do. This combination of configs will send uncooked, cloned data, to different indexer groups (or individual indexers).

On your intermediate forwarder, configure the following under $SPLUNK_HOME/etc/system/local/:

props.conf:
[your:sourcetype:name]
TRANSFORMS-routing = cloneAll

transforms.conf:
[cloneAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Group1,Group2

outputs.conf:
[tcpout]
defaultGroup=nothing

[tcpout:Group1]
disabled=false
sendCookedData=false
server=10.x.xx.1:9997

[tcpout:Group2]
disabled=false
sendCookedData=false
server=10.x.xx.2:1234

Forgot to mention, cycle Splunk to implement the changes.

Re: Sending uncooked data from indexer level

gyarici — Wed, 12 Feb 2020 11:35:43 GMT

Thanks for the answer. So, i need to send uncooked data from IF to indexer first (?) Anyhow, i need to send uncooked data directly from indexers to 3rd party app ! I think it should be performed before indexer parsing

Re: Sending uncooked data from indexer level

gyarici — Wed, 12 Feb 2020 11:38:21 GMT

Thanks for the information. I think if there is no way, this should be enhancement point as many of the architectures have tones of UF/IF and it is not practical/efficient to use UF to send the data to 3rd party apps

Re: Sending uncooked data from indexer level

codebuilder — Wed, 12 Feb 2020 15:12:44 GMT

That's not true. Assuming you do not want or need to do any parsing at the forwarder level, you can accomplish this same task with a UF. See my second answer.

Re: Sending uncooked data from indexer level

codebuilder — Wed, 12 Feb 2020 15:15:31 GMT

To accomplish the same data cloning with a universal forwarder, you can use the example below.

 outputs.conf:
 [tcpout]
 defaultGroup=Group1,Group2

 [tcpout:Group1]
 disabled=false
 sendCookedData=false
 server=10.x.xx.1:9997

 [tcpout:Group2]
 disabled=false
 sendCookedData=false
 server=10.x.xx.2:1234

Re: Sending uncooked data from indexer level

gyarici — Thu, 13 Feb 2020 09:08:22 GMT

Thanks alot for the response.

Ok. Then could you please clarify how I can configure indexers to send uncooked data to the 3rd party applications/servers?

Send uncooked data:

1.Intermediate Forwarder-> 2.Indexer -> 3.3rd party server

I assumed between step 1 and 2 above are ok with your configuration. I need to know more about between step and step 3 configuration which i should implement on indexers.

Thanks

Re: Sending uncooked data from indexer level

codebuilder — Thu, 13 Feb 2020 17:00:17 GMT

Sending data directly from an indexer to a 3rd party system is not good practice nor a good idea. The supplied solutions are the accepted method to accomplish what you're trying to do.