Installation

CPU usage of splunkd processes hangs to 100% after upgrade to Splunk 6

mas
Path Finder

Hi,

after the upgrade to Splunk 6 from Splunk 4.3.3, we have serious problems with our single-server instance (Windows Server 2008 R2). In fact during the night the CPU usage of "splunkd" processes reaches 100% and hangs at this level, making the server unreachable via WEB interface, but often also via RDP connections. The only solution is to terminate the splunkd process and then restart it.

Having a look at splunkd.log, we see a continuous flooding of these errors (over 35.000 errors per second):

ERROR TcpChannel - Error trying to begin socket accept: An invalid argument was supplied.

These errors disappear after splunkd service has been restarted.

We guess there is some problem with inbound connections from Universal Forwarders (that have not been upgraded yet), but we have no clue to confirm this diagnosis. System event logs do not report any warning or error.

Any suggestions would be really appreciated.

Regards.

0 Karma

jeandez
Explorer

i forgot to mention that i am using Splunk 6

0 Karma

jeandez
Explorer

I have the same problem. I am using splunk on a 12 cpu and 8 Go RAM system. when i run top command i notice 233 % of CPU use by splunkd.
Please i need more information about this process.
How splunkd works in a multi processor system ?

kind regards

0 Karma

davidpaper
Contributor

What's the last thing you see in the splunkd.log before the ERROR TcpChannel messages?

0 Karma

mas
Path Finder

Unluckily I have now way to identify the error that occurred immediately before the TcpChannel messages, because these are generated too fast and the splunkd.log files are flooded with them. The only error we can see in the splunkd.log and its associated rotation files is the TcpChannell message, repeated indefinitely.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...