Splunk sourcetype top to capture memory in terabyte

[harry521]

Splunk has a top sourcetype which can help to monitor the system resource usage. I recently ran into a problem while the RH7 outputs RES in terabyte(t) while process is over 10G of memory usage. The top output in Splunk is in KB as what I understand, and converts MB, GB correctly, but not TB. I had looked into the top script and sourcetype. Find no clue how MB or GB is converted. Any solution?

[harry521]

I actually found an answer for myself and it's simple. Instead of using "top" for memory, I switched to "ps". And the column "RSZ_KB" is the "RES" from top output. No more issue when memory go over 10G.

[harry521]

I looked into it a little bit more. I found out that might be something related to RH 7. I have RH 6 being monitored and that works well. For example: On both RH7 and 6, if RES is under 10G, it will be output the value converted to KB, like 10,000,000. However, on RH7, when it's above 10 G, it will be converted to TB like 0.01 and so on. This messed up my memory time chart.

I'm using splunk 6.5. Is there possibly a newer version has a patch or update of the top.sh script?

[niketn]

@harry521, what is the current query that you are running?

[harry521]

simply execute ./bin/top.sh every x sec and search for sourcetype=top.

index=os sourcetype=top COMMAND="java"