Monitor CPU cores of a Linux machine

[hishamjan]

Hi, 

I have a Linux machine running on Centos 6.10 with a quad-core processor  (16 threads)

On Splunk, is there a way to monitor each of its threads? Because sometimes, the threads reach 100% and my system stops working until the threads are down to the current percentage values as in the above screenshot.

Any degree of help will be appreciated.

Regards.

[jokertothequinn]

This will do the trick:

| mstats avg(cpu_metric.*) as cpu_* WHERE index=<your_metrics_index> by CPU, host
| table CPU, host
| eventstats max(CPU) as cpu_count by host
| table cpu_count, host
| eval cpu_count=cpu_count+

the data being used is from the add on Link to the splunk add on for Splunk Add-on for Unix and Linux docs 

[jokertothequinn]

This is a query which can get you the cpu core for both nix and win servers:

| mstats avg(Processor.*) as * WHERE (index=win-metrics) instance!="_Total" host="***"  by host instance span=5m
| table _time, host, instance, "%_Processor_Time"
| stats max(instance) as "cpu_core" by host
| eval cpu_core=cpu_core + 1
| append
[| mstats avg(cpu_metric.*) as cpu_* WHERE (index=nix-metrics) host="***" by CPU, host
| table CPU, host
| eventstats max(CPU) as cpu_core by host
| stats max(cpu_core) as cpu_core by host
| eval cpu_core=cpu_core + 1 ]

[gcusello]

Hi @hishamjan,

yes, you can monitor everything,

You can use the Splunk AddOn for Unix and Linux (https://splunkbase.splunk.com/app/833/) to take the logs and the Splunk App for Unix and Linux (https://splunkbase.splunk.com/app/273/) to display information.

If instead you want a custom control, you can create a custom scripted input using the command you used for the screenshot (https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptedInputsIntro) to index the values and you can crete your own dashboard.

Anyway I hint to try with the splunkbase apps before to start to develop a custom one.

Ciao.

Giuseppe

[hishamjan]

hi @gcusello 

sorry for the late response, but my Linux machine already has Splunk AddOn for Unix installed.

Is it possible to monitor its cores (threads)?

Thanks

[gcusello]

Hi @hishamjan,

in the Splunk AddOn for Unix there are some scripts that run Linux commands as  cpu or iostat or others: they, when enabled, take stats on the cpu utilization.

I don't know the command you used in your screenshot, but if it's one of the already presente in app scripts you can take results only enabling it.

If instead the command you used is another one, you can see how scripts are done in that app and create your own scripted input using your command.

Anyway, in few words:

• create in the $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin the file your_command.sh;
• in the your_command.sh file put the command with all the options you want;
• enable it to be executed;
• create a stanza in inputs.conf (the one in local folder not in default) defining the frequency execution;
• restart splunk on the machine or deploy it using the Deployment Server.

the script is very easy, it contains only the command as you run it in SSH.

the inputs.conf stanza is something like this:

[script://./bin/your_command.sh]
sourcetype = cpu_usage
source = your_command
interval = 300
disabled = 0

Ciao.

Giuseppe