High swap space observed in HF ?

[narisree1]

We are getting alert from our LInux team stating high swap space observed for splunkd process on the Heavy forwarder which also acts as a syslog server .

Below command is said to be consuming

splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd

PID      USER PR  NI  VIRT           RES        SHR     S      %CPU  %MEM   TIME+     COMMAND
5580 splunk 20 0 4966352 878220 15516  S         57.8         5.4     5028:07   splunk

SWAP TOTAL = 4617084928 bytes
SWAP USED = 4617084928 bytes

Please let me know what should be done to fix this issue .

[narisree1]

Well there is 1 TB of space and only 9 % is used with respective to the storage space . Issue observed is that the CPU usage goes beyond 100% and memory space utilization is high  . 

[richgalloway]

Using swap is not abnormal.  A process that needs a lot of memory to handle a lot of data will swap when it needs more memory.

When you say "HF which also acts as a syslog server" does that mean syslog events are received on port 514 of the HF?  If so, that's sub-optimal.  Use a real, dedicated syslog server (like rsyslog or syslog-ng) with a universal forwarder.  Even better is the Splunk Connect for Syslog (SC4S) app, which is a Docker app that receives syslog and forwards it to HEC inputs.