Monitoring Splunk

High swap space observed in HF ?

narisree1
Loves-to-Learn Everything

We are getting alert from our LInux team stating high swap space observed for splunkd process on the Heavy forwarder which also acts as a syslog server .

Below command is said to be consuming

splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
 
PID      USER PR  NI  VIRT           RES        SHR     S      %CPU  %MEM   TIME+     COMMAND
5580 splunk 20 0 4966352 878220 15516  S         57.8         5.4     5028:07   splunk
 
SWAP TOTAL = 4617084928 bytes
SWAP USED = 4617084928 bytes
 
Please let me know what should be done to fix this issue .
Labels (1)
0 Karma

narisree1
Loves-to-Learn Everything

Well there is 1 TB of space and only 9 % is used with respective to the storage space . Issue observed is that the CPU usage goes beyond 100% and memory space utilization is high  . 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Using swap is not abnormal.  A process that needs a lot of memory to handle a lot of data will swap when it needs more memory.

When you say "HF which also acts as a syslog server" does that mean syslog events are received on port 514 of the HF?  If so, that's sub-optimal.  Use a real, dedicated syslog server (like rsyslog or syslog-ng) with a universal forwarder.  Even better is the Splunk Connect for Syslog (SC4S) app, which is a Docker app that receives syslog and forwards it to HEC inputs.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.