Getting Data In

Can a Heavy Forwarder send cooked but unparsed data?

lbur
Explorer

Is it possible to have a heavy forwarder send unparsed (not raw) cooked data?
I have a server which needs to forward data, and a universal forwarder sending compressed, unparsed data would be fine.
However, I would like to use that same server to do some data collection as well.
This data collection requires a full Splunk install and a 3rd party app (estreamer to be specific).
However, as I understanding it using a full Splunk install as a heavy forwarder will, by default send parsed data.
This is a much heavier network load, which I would like to avoid.
The only option in outputs.conf related to this is: sendCookedData = true | false.
If I set this to false, then it will be sending raw (uncooked data to the forwarder).
If I set this to true, then it appears the heavy forwarder will send all data as cooked, parsed data.
I'm looking for an option to send cooked, unparsed data.

Thanks for any help!

0 Karma

ekost
Splunk Employee
Splunk Employee

Hi folks. The easiest way to minimize network bandwidth impact from a HF is to:
1. Use the HF to monitor/interface with only the data sources that you need the HF for. I assume you want the HF for the UI needs, but some apps might use the parsing. This is what @somesoni2 is recommending.
2. Send the cooked, parsed output to the indexers via SSL, leveraging the SSL compression. This dramatically lowers the network impact, in exchange for admin overhead of setup and configuration.

0 Karma

somesoni2
Revered Legend

One option would to have both Splunk Enterprise Instance and a Universal Forwarder on your machine generating data. Use Heavy forwarder only for estreamer specific monitoring and UF for rest. (you can't turn off parsing on HF, you may configure it to be reparsed at indexers).

0 Karma

lbur
Explorer

Thanks Koshyk, but that didn't answer the question. I want the Heavy Forwarded to send unparsed cooked data to the indexers. You addressed sending raw and parsed.

0 Karma

koshyk
Super Champion

You could have ANY number of outputs to any locations. So you send outputs to "Destination1 - Third Party" which is not cooked, but to "destination2 - xyz" which is cooked etc. Each tcpout stanza can vary depending on how you want it.
( by-the-way estreamer is a pain !! and I won't go near it as it is unsupported)

Also the heavy-forwarder gives option for OUTPUT in syslog format which is great to way to make Splunk work as a logging engine to centralised logging solutions. In one of the customers, we collect using Splunk UF from various machines and at Heavy-forwarder we dump to syslog server for multiple other third parties. We don't want to integrate , but rather we dump into central location in uncooked format and it is up-to the company/third-party how they take it.

0 Karma

lbur
Explorer

Thanks Koshyk, but that didn't answer the question. I want the Heavy Forwarded to send unparsed cooked data to the indexers. You addressed sending raw and parsed.

0 Karma

koshyk
Super Champion

I'm assuming you want to send from UF to a full Splunk installation.
If you look into the indexing piplelines , the UF does NOT do the real parsing (Detail Diagram - UF/LWF to Indexer). So the output from from ur UF will be "cooked but unparsed data"

0 Karma

lbur
Explorer

No, I want to send from a Heavy Forwarder (because I need the full Splunk Installation for other purposes, like estreamer and dbconnect), but for normal file monitor functions (for instance) I want to forward cooked, unparsed data, in order to limit the network bandwidth. However, it seems as if the Heavy Forwarder can only be configured to send either raw, or cooked parsed data, which is much larger than unparsed data.

0 Karma

danielhernandez
New Member

Hi lbur, I'm having the same problem. Trying to send cooked and unparsed data from a Heavy Forwarder, so we don't have to re-distribute or re-plan data collection/ingestion, and I'm trying to avoid any additional configuration on the indexers' queues.

Have you gotten anywhere with this?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.