Getting Data In

Why does an AIX 6.5.2 forwarder have high swap/memory and cpu consumption?

ddrillic
Ultra Champion

We see the following -

sh-4.2$ ps avwx | head -1; ps avwx | sort +4n -r | head -10
      PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
  7274610      - A    51121:15 3427531 1739848 749560    xx 100303  9692  1.0  8.0 splunkd -p 8089 start

What can it be?

Tags (2)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

I suspect you are misinterpreting the stats if the question is correct, I would suggest you use svmon in AIX to accurately determine the memory in use.
Reading your question you appear to be using 1% CPU.

Here's how to measure memory use in AIX:
svmon -P 7274610 -O unit=MB

I checked two production forwarders, a 6.5.2 instance was:
Pid Command Inuse Pin Pgsp Virtual
22937622 splunkd 1570.29 39.6 0 300.84

Another instance I checked 7.0.0:
Pid Command Inuse Pin Pgsp Virtual
57540776 splunkd 1185.07 256.23 4.55 558.29

Both show 1% CPU in the ps command, you might like to open topas in AIX and see if you are seeing high CPU by Splunk.

If you are looking for a more comprehensive monitoring solution I use the Nmon application for Splunk on AIX servers (and Linux) , the official Splunk add on for unix is here and the app is here

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

I suspect you are misinterpreting the stats if the question is correct, I would suggest you use svmon in AIX to accurately determine the memory in use.
Reading your question you appear to be using 1% CPU.

Here's how to measure memory use in AIX:
svmon -P 7274610 -O unit=MB

I checked two production forwarders, a 6.5.2 instance was:
Pid Command Inuse Pin Pgsp Virtual
22937622 splunkd 1570.29 39.6 0 300.84

Another instance I checked 7.0.0:
Pid Command Inuse Pin Pgsp Virtual
57540776 splunkd 1185.07 256.23 4.55 558.29

Both show 1% CPU in the ps command, you might like to open topas in AIX and see if you are seeing high CPU by Splunk.

If you are looking for a more comprehensive monitoring solution I use the Nmon application for Splunk on AIX servers (and Linux) , the official Splunk add on for unix is here and the app is here

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

MuS
SplunkTrust
SplunkTrust

Try accessing this REST endpoint on your UF https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus to see how may files are being monitored. High numbers of monitored files can cause such behaviour ...

ddrillic
Ultra Champion

@MuS - only two files are being monitored ...

0 Karma

MuS
SplunkTrust
SplunkTrust

How many directories needs to be scanned by the UF to reach those two files? Also can you try truss the process and see what it actually does?

0 Karma

ddrillic
Ultra Champion

Barely five directories and explicit two files to monitor ; - ) maybe an AIX specific issue?

0 Karma

MuS
SplunkTrust
SplunkTrust

Actually looking at the numbers are 1% CPU and 8% memory usage really that high? Does vmstat provide some hints where the potential bottleneck could be?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!