Splunk Search

Splunk search to find the disk utilization on multiple servers

rcreddy06
Path Finder

I've 5000 linux servers and I would like to do a splunk search to get their disk utilization. Its not possible to do a df on 5000 servers, I'm doing a dashboard for servers that crosses 85% utilization and schedule a PDF delivery to my email. What would be the best splunk search command to find the disk utilization %'s for splunk on all the servers and filter to servers greater than 85%.

1 Solution

jkat54
SplunkTrust
SplunkTrust

Lets assume your data looks like this:
host=server1,dev=/dev/sda1,free=1000,used=1000,total=2000

You would run this search:
index=indexname | eval utilization=(used/total)*100 | stats first(utilization) by host | where utilization > 84

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

Lets assume your data looks like this:
host=server1,dev=/dev/sda1,free=1000,used=1000,total=2000

You would run this search:
index=indexname | eval utilization=(used/total)*100 | stats first(utilization) by host | where utilization > 84

0 Karma

jkat54
SplunkTrust
SplunkTrust

I wouldnt call this the "best" search but it's certainly a search that would work.

0 Karma

lcrielaa
Communicator

By installing the Splunk app for *nix, and deploying the *nix TA to your Linux servers, you can have your servers report the output of the "df" command in Splunk. You can then do a search on your collected "df" data to find servers that have 85% or higher disk utilisation.

For this you would need:
A Splunk server (search head / indexer)
A Splunk Deployment server to push out the *nix TA (Doesn't need to be a separate server)
A Splunk universal forwarder installed on your 5000 Linux servers that is connected to your deployment server.

Do you have any of this in place yet? Secondly, are you sure Splunk is the best answer for your problem(s)? You don't have any performance metrics collection in place already?

0 Karma

jeffland
SplunkTrust
SplunkTrust

Just to clarify, splunk is not some sort of "job distribution engine". You don't enter a command in splunk which is then sent to connected servers.
Instead, you define inputs on the servers, and they send their output to splunk. You can then search on and interpret that data (or create alerts and email delivery) as much as you like.

0 Karma

rcreddy06
Path Finder

I was referring to search query to get results and create dashboards.

0 Karma

jeffland
SplunkTrust
SplunkTrust

And I wanted to point out that you will have to make all 5000 servers run a df once in a while and capture the output of that with splunk in order to be able to put that info on a dashboard.
If you don't want to do all that by hand, you might be interested in this app/TA.

lguinn2
Legend

What version of Splunk are you running?
So what is the data that you have collected in Splunk from the 5000 servers? Because you can't write a dashboard if you don't have the data...

0 Karma

rcreddy06
Path Finder

Version 6.3. The universal forwarders sends the syslog data and other oracledb data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...