Why did LINE_BREAKER on a Splunk 6.1.1 universal forwarder cause a CPU spike?

[a212830]

Hi,

I had a customer complaining that the Universal Forwarder on their server was running very hot. I checked, and lo and behold, it was running at 100% (Splunk 6.1.1). I checked the splunkd.log, and it had some errors:

Line breaking regex has no capturing groups: ^\d{2}:\d{2}:\d{2}\.\d{3} 

Now, I thought the UF didn't actually break the feed into events. If so, why would this error cause the cpu spike in such a manner? I adjusted the line to be:

LINE_BREAKER = ([\r\n]+)[0-2][0-9]:[0-5][0-9]:[0-5][0-9]\.\d{2,5}\s

And, the spike went way.

[sloshburch]

Oooo, this is an interesting one.

So two thoughts on this:

1. Parsing at the UF: It is interesting that the parsing phase was being invoked. Especially given that we know from http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F that parsing should not be occurring on the UF. Alternatively, there is mention on that page that as of 6.0, some parsing could occur at the endpoint (esp for structured sources). It's hard to know for sure without seeing the rest of that sourcetype's configuration (props and transforms). Feel free to share a sanitized version for extra context.
2. CPU Spike: It sounds like you already know, but for others, it seems the spike was the result of the fact that the forwarder was doing a ton of work trying to send across massive events since there was no breaking going on. I think the forwarders send like 32k payloads or something but I don't recall how they split those payloads if the events are in fact being parsed and split on the UF.

Were you able to grab a diag? If you are able to reproduce on a lab box and grab a diag then I would encourage you to open a help desk ticket. Those guys will more easily be able to see where in the pipeline this got caught and if you caught some bug in the parsing logic.