topic Re: create dashboard for live monitoring for cpu, disk space & memory in Dashboards & Visualizations

How to create dashboard for live monitoring for cpu, disk space & memory?

nikhilmfwd — Fri, 17 Feb 2023 15:37:09 GMT

Dears,

I have installed Splunk app for linux & add on in my Splunk enterprise paid license version. Installed splunk forwarder in all hosts & added cpu, vmstat & df in input.conf file in remote servers. Now i want to create dashboard for live monitoring for mentioned linux metrics & alerts for that.

Need to help to do that or have any good documents please share.

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Fri, 17 Feb 2023 07:46:50 GMT

Hi @nikhilmfwd,

at first how did you enabled inputs.conf? manually or using the Splunk_TA_nix (https://splunkbase.splunk.com/app/833 )?

if manually, use te above add-on.

Then see in splunkbase if there's some linux app that contains the dashboards you want.

some examples are:

https://splunkbase.splunk.com/app/3702

https://splunkbase.splunk.com/app/3777

https://splunkbase.splunk.com/app/6702

otherwise you could try this dashboard that I did some years ago:

<form> <label>Hardware and Software Details: Linux Servers</label> <fieldset submitButton="false"> <input type="dropdown" token="host"> <label>Server</label> <prefix>host="</prefix> <suffix>"</suffix> <fieldForLabel>host</fieldForLabel> <fieldForValue>host</fieldForValue> <search> <query>index=os sourcetype=hardware | eval host=upper(host) | dedup host | sort host | table host</query> </search> </input> </fieldset> <row> <panel> <title>HostName</title> <html> <h3 align="center"> <strong> <font size="10">Server<img src="/static/app/infrastructure_monitoring/Linux_logo.png" style="height:100px;border:0;"/> </font> </strong> </h3> </html> <single> <search> <query>index=os sourcetype=hardware $host$ | dedup host | table host</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> </single> </panel> </row> <row> <panel> <title>Hardware</title> <table> <search> <query>index=os sourcetype=hardware $host$ | dedup host | eval MEMORY_REAL=MEMORY_REAL/1024/1024, MEMORY_SWAP=MEMORY_SWAP/1024/1024, host=upper(host) | lookup Server host OUTPUT IP Tipologia | table IP Tipologia CPU_TYPE CPU_COUNT CPU_CACHE MEMORY_REAL MEMORY_SWAP fd0 hdc sda | rename CPU_TYPE AS CPU CPU_COUNT AS "Number of CPUs" CPU_CACHE AS Cache MEMORY_REAL As RAM MEMORY_SWAP AS Swap HARD_DRIVES AS "Hard Disks" fd0 AS "Floppy Disk" hdc AS "Hard Disk" sda AS "Virtual disk"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> <panel> <title>df</title> <table> <search> <query>index=os sourcetype=df $host$ | dedup host | multikv | table Filesystem Type Size Used Avail UsePct MountedOn</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> </row> <row> <panel> <title>Processes</title> <table> <search> <query>index=os sourcetype=ps $host$ | multikv | table USER PID PSR pctCPU CPUTIME pctMEM RSZ_KB VSZ_KB TTY S ELAPSED COMMAND ARGS</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> <panel> <title>top command</title> <table> <search> <query>index=os sourcetype=top $host$ | dedup host | multikv | table PID USER PR NI VIRT RES SHR S pctCPU pctMEM cpuTIME COMMAND</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> </row> <row> <panel> <title>netstat</title> <table> <search> <query>index=os sourcetype=netstat $host$ | dedup host | multikv | table Proto Recv-Q Send-Q LocalAddress ForeignAddress State</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> <panel> <title>packages</title> <table> <search> <query>index=os sourcetype=package $host$ | multikv | dedup host NAME | table NAME VERSION RELEASE ARCH VENDOR GROUP | sort NAME</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> </row> <row> <panel> <title>openPorts</title> <table> <search> <query>index=os sourcetype=openPorts $host$ | dedup host | multikv | table Proto Port</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> <panel> <title>protocol</title> <table> <search> <query>index=os sourcetype=protocol $host$ | dedup host | multikv | table IPdropped TCPrexmits TCPreorder TCPpktRecv TCPpktSent UDPpktLost UDPunkPort UDPpktRecv UDPpktSent</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> </row> <row> <panel> <title>Users with private logins</title> <table> <search> <query>index=os sourcetype=usersWithLoginPrivs $host$ | dedup host | multikv | table USERNAME HOME_DIR USER_INFO</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Floppy Disk"> <option name="unit">GB</option> </format> <format type="number" field="Hard Disk"> <option name="unit">GB</option> </format> <format type="number" field="Virtual disk"> <option name="unit">GB</option> </format> <format type="number" field="RAM"> <option name="unit">GB</option> </format> <format type="number" field="Swap"> <option name="unit">GB</option> </format> <format type="number" field="Cache"> <option name="unit">kB</option> </format> </table> </panel> </row> </form>

Ciao.

Giuseppe

Re: create dashboard for live monitoring for cpu, disk space & memory

nikhilmfwd — Fri, 17 Feb 2023 07:52:11 GMT

Dear Sir,

I have enabled inputs.conf using the Splunk_TA_nix, inside

/apps/splunkforwarder/etc/apps/Splunk_TA_nix/local/input.conf added mentioned things for getting data in all my remote servers.

echo -e "[script://./bin/vmstat_metric.sh]

sourcetype = vmstat_metric

source = vmstat

index=linux

interval = 60

disabled = 0

[script://./bin/df_metric.sh]

sourcetype = df_metric

source = df

index=linux

interval = 300

disabled = 0

[script://./bin/cpu_metric.sh]

sourcetype = cpu_metric

source = cpu

index=linux

interval = 30

disabled = 0

[script://./bin/vmstat.sh]

interval = 60

sourcetype = vmstat

source = vmstat

index=linux

disabled = 0

[script://./bin/df.sh]

interval = 300

sourcetype = df

source = df

index=linux

disabled = 0

[script://./bin/cpu.sh]

sourcetype = cpu

source = cpu

interval = 30

index=linux

disabled = 0 " > /apps/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Fri, 17 Feb 2023 07:56:09 GMT

Hi @nikhilmfwd,

the TA is correct, see the other apps.

Ciao.

Giuseppe

Re: create dashboard for live monitoring for cpu, disk space & memory

nikhilmfwd — Fri, 17 Feb 2023 08:03:08 GMT

Hi,

I didnt get that. I need to see other apps for what?

can in create dashboard from these data? using Splunk App for Unix?

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Fri, 17 Feb 2023 08:59:05 GMT

Hi @nikhilmfwd,

you can display data from linux servers using your search or the dashboard I shared or see in the listed apps if there's some other dashboard that can be useful for you.

Ciao.

Giuseppe

Re: create dashboard for live monitoring for cpu, disk space & memory

nikhilmfwd — Fri, 17 Feb 2023 09:02:58 GMT

hi @gcusello sir,

Thanks for the help.!!

For sure it will be more useful for me. I will try to create dashboard if any issue i ll get back to you.

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Fri, 17 Feb 2023 09:30:45 GMT

Hi @nikhilmfwd,

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Fri, 17 Feb 2023 10:03:37 GMT

Hi @nikhilmfwd ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: create dashboard for live monitoring for cpu, disk space & memory

kevhead — Tue, 06 Feb 2024 16:40:03 GMT

@gcusello Thank you for providing this code for the dashboard. I've implemented it and its working quite well except for the hardware portion which returns a " Error in 'lookup' command: Could not construct lookup 'Server, host, OUTPUT, IP, Tipologia'. See search.log for more details". Any assistance with this would be great thank you!

Re: create dashboard for live monitoring for cpu, disk space & memory

gcusello — Tue, 06 Feb 2024 16:59:39 GMT

Hi @kevhead ,

sorry, it was a mistyping: in that installation I had a lookup containg some additional informa that you can delete from the dashboard.

Ciao.

Giuseppe