sendCookedData=false causing message rejects

[responsys_cm]

I have a customer where the Splunk team does not have management access to forwarders and the ops people won't allow agents to be managed by a deployment server. Their data is kind of messy and requires a number of sourcetype and host metadata rewrites.

Since pushing out any changes to the forwarders is a slow, time consuming process, it makes sense to put the metadata rewrites and routing logic on the indexers. This would require that UFs and intermediate forwarders have sendCookedData=false in their outputs.conf file.

When I enabled that setting on the UFs and intermediate forwarders, data stopped flowing in and I saw a ton of the following messages:

07-31-2018 17:47:02.429 +0000 ERROR TcpInputProc - Message rejected. Received unexpected message of size=1249209376 bytes from src=10.192.1.7:64398 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

I saw this on both the intermediate forwarders and the indexers. I looked for "67108864" in the default limits.conf, but couldn't find anything.

Anyone know how to disable cooked data without triggering this message?

Thx.

C

[hettervik]

Hi C! Did you find out the exact reason for the error messages? I've encountered the same issue when trying to send uncooked data from my deployment server (DS) to my indexer. The strange thing is that even the internal Splunk data from my DS stopped flowing, even though the log lines of said data should be quite small.

[MuS]

Hi responsys_cm,

You are entering danger zone, here:

You can parse cooked data again, without any problem. You just need to change one little setting, you can read more about it here : https://answers.splunk.com/answers/168491/routing-data-to-index-using-sourcetype.html#comment-168781
Also, pay attention to the remark made by @jrodman !

I would also recommend adding a dedicated heavy weight forwarder to do the re-parsing of the events, instead doing it on the indexers.

Another option is to wait until after .conf18 - hint https://conf.splunk.com/learn/session-catalog.html?search=FN1919#/ 😉

Hope this helps ...

cheers, MuS

[responsys_cm]

Thanks, MuS... It's not a danger zone I wanted to wade into. I have access to my customer's cluster and intermediate forwarders, but not the UFs. Looks like the smart move is to push those transforms to the UFs and do it there...