Why am I seeing a lot of name=cooked_output events in _internal?

[vicvaughan]

All of a sudden, noticed getting tons of events in _internal with name=cooked_output. What could be causing this behavior?

[craigv_splunk]

This event is logged when Splunk sends data over the network. Data can be sent in two ways 1) cooked-when it is sent to another splunk instance and 2)uncooked-when it is being sent somewhere else like to a syslog server. My guess is this the log of a universal forwarder which is sending data to a Splunk indexer. The number of these events will scale proportional to the universal forwarders. They are benign and not a cause for concern. They are provided for informational reasons

[vicvaughan]

Thanks for your answer, Craig. The thing that was troubling is that starting on June 18, we have gone from about 10 of these per day to around 1-2 million cooked_output events per day.

[craigv_splunk]

Interesting. Is there anything that happened on the day that changed in your infrastructure i.e Splunk upgrades, new hosts, major config changes?

[vicvaughan]

No. We've talked to the infrastructure guys and the last patches were before the behavior started by a month or so. So the number of cooked seemed excessively high comparatively.

[craigv_splunk]

Hmm. If there are no Warn or Error messages I don't think that it is anything benign for now. It could be caused by changes in the logging behavior/frequency of a particular log that is being monitored. In my opinion, keep an eye on your environment for more WARN or ERROR messages that would be a clearer indicator that something is wrong.

[craigv_splunk]

Could you perhaps post the entire event line with source and sourcetype information?

[vicvaughan]

yes...

This is typically how it looks:

07-19-2016 15:18:53.994 -0500 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.300961, instantaneous_eps=0.354828, average_kbps=0.397792, total_k_processed=489797.000000, kb=9.330078, ev=11.000000