Why does an AIX 6.5.2 forwarder have high swap/memory and cpu consumption?

[ddrillic]

We see the following -

sh-4.2$ ps avwx | head -1; ps avwx | sort +4n -r | head -10
      PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
  7274610      - A    51121:15 3427531 1739848 749560    xx 100303  9692  1.0  8.0 splunkd -p 8089 start

What can it be?

[gjanders · Accepted solution]

I suspect you are misinterpreting the stats if the question is correct, I would suggest you use svmon in AIX to accurately determine the memory in use.
Reading your question you appear to be using 1% CPU.

Here's how to measure memory use in AIX:
svmon -P 7274610 -O unit=MB

I checked two production forwarders, a 6.5.2 instance was:
Pid Command Inuse Pin Pgsp Virtual
22937622 splunkd 1570.29 39.6 0 300.84

Another instance I checked 7.0.0:
Pid Command Inuse Pin Pgsp Virtual
57540776 splunkd 1185.07 256.23 4.55 558.29

Both show 1% CPU in the ps command, you might like to open topas in AIX and see if you are seeing high CPU by Splunk.

If you are looking for a more comprehensive monitoring solution I use the Nmon application for Splunk on AIX servers (and Linux) , the official Splunk add on for unix is here and the app is here

[MuS]

Try accessing this REST endpoint on your UF https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus to see how may files are being monitored. High numbers of monitored files can cause such behaviour ...

[ddrillic]

@MuS - only two files are being monitored ...

[MuS]

How many directories needs to be scanned by the UF to reach those two files? Also can you try truss the process and see what it actually does?

[ddrillic]

Barely five directories and explicit two files to monitor ; - ) maybe an AIX specific issue?

[MuS]

Actually looking at the numbers are 1% CPU and 8% memory usage really that high? Does vmstat provide some hints where the potential bottleneck could be?