Monitoring Splunk

Why am I seeing a lot of name=cooked_output events in _internal?

vicvaughan
Explorer

All of a sudden, noticed getting tons of events in _internal with name=cooked_output. What could be causing this behavior?

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

This event is logged when Splunk sends data over the network. Data can be sent in two ways 1) cooked-when it is sent to another splunk instance and 2)uncooked-when it is being sent somewhere else like to a syslog server. My guess is this the log of a universal forwarder which is sending data to a Splunk indexer. The number of these events will scale proportional to the universal forwarders. They are benign and not a cause for concern. They are provided for informational reasons

0 Karma

vicvaughan
Explorer

Thanks for your answer, Craig. The thing that was troubling is that starting on June 18, we have gone from about 10 of these per day to around 1-2 million cooked_output events per day.

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

Interesting. Is there anything that happened on the day that changed in your infrastructure i.e Splunk upgrades, new hosts, major config changes?

0 Karma

vicvaughan
Explorer

No. We've talked to the infrastructure guys and the last patches were before the behavior started by a month or so. So the number of cooked seemed excessively high comparatively.

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

Hmm. If there are no Warn or Error messages I don't think that it is anything benign for now. It could be caused by changes in the logging behavior/frequency of a particular log that is being monitored. In my opinion, keep an eye on your environment for more WARN or ERROR messages that would be a clearer indicator that something is wrong.

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

Could you perhaps post the entire event line with source and sourcetype information?

0 Karma

vicvaughan
Explorer

yes...

This is typically how it looks:

07-19-2016 15:18:53.994 -0500 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.300961, instantaneous_eps=0.354828, average_kbps=0.397792, total_k_processed=489797.000000, kb=9.330078, ev=11.000000
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!