Monitor CPU Usage, RAM Usage, Hard Disk Utilization, Load Average, Largest Files and LAN Card Traffic

[hishamjan]

Hi,

I'm running Splunk Enterprise v7.0.1 (Indexer) on a separate Linux server with Splunk Forwarders on two more Linux servers that are forwarding data to the Indexer.

I would like to monitor;

1)CPU Usage,

2)RAM Usage

3)Hard Disk Utilization

4)Load Average

5)Largest Files

6)LAN Card Traffic

The monitoring Console on the Indexer fails to show these metrics for all other instances bar its local.

Is there any way to monitor these metrics for the forwarders as well as the localhost?

I appreciate whoever is willing to help.

Thanks and regards.

[gcusello]

Hi @hishamjan,

to do this you should take the logs from the Forwarders using the Splunk Add-on for Unix and Linux (https://splunkbase.splunk.com/app/833/) that already has all the inputs to measure the parameters you want.

Then you can create your own dashboards using the logs from the above Add-on or install the Splunk App for Linux and Unix (https://splunkbase.splunk.com/app/273/) that contains all the dashboard you need.

The third solution is to analyze the above App taking only the searches you need for your Use Cases, eventually customizing them.

Ciao.

Giuseppe

[hishamjan]

Hi Guiseppe,

I have already installed the Add-on for Unix and Linux and that's where I am able to produce a working environment. However, what I don't know is how to measure the parameters that you're suggesting already exist in the add-on.

Kindly help me with an elaborated response on that.

Thanks for your kind help.

Regards,

Hisham

[gcusello]

Hi @hishamjan,

this is a dashboard that i used for Linux Servers monitoring:

<form>
  <label>Hardware and Software Details: Linux Servers</label>
  <fieldset submitButton="false">
    <input type="dropdown" token="host">
      <label>Server</label>
      <prefix>host="</prefix>
      <suffix>"</suffix>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>
          index=os sourcetype=hardware 
          | eval host=upper(host) 
          | dedup host 
          | sort host 
          | table host
        </query>
      </search>
    </input>
  </fieldset>
  <row>
    <panel>
      <single>
        <title>HostName</title>
        <search>
          <query>
            index=os sourcetype=hardware $host$ 
            | eval host=upper(host)
            | dedup host 
            | table host
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
      </single>
    </panel>
      <single>
        <title>Description</title>
        <search>
          <query>
            index=os sourcetype=hardware $host$
            | eval host=upper(host)
            | lookup Perimeter.csv Hostname AS host OUTPUT Description 
            | table Description
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Hardware</title>
      <table>
        <search>
          <query>
            index=os sourcetype=hardware $host$
            | dedup host 
            | eval MEMORY_REAL=MEMORY_REAL/1024/1024, MEMORY_SWAP=MEMORY_SWAP/1024/1024, host=upper(host)
            | table CPU_TYPE CPU_COUNT CPU_CACHE MEMORY_REAL MEMORY_SWAP fd0 hdc sda 
            | rename CPU_TYPE AS CPU CPU_COUNT AS "Number of CPUs" CPU_CACHE AS Cache MEMORY_REAL As RAM MEMORY_SWAP AS Swap HARD_DRIVES AS "Hard Disks" fd0 AS "Floppy Disk" hdc AS "Hard Disk" sda AS "Virtual disk"
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">100</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
    <panel>
      <title>Operative System</title>
      <table>
        <search>
          <query>
            index=os sourcetype=Unix:Version $host$
            | table os_name os_release os_version machine_architecture_name
            | rename os_name AS "Operative System" os_release AS Release os_version AS Version machine_architecture_name AS Architecture
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">100</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>df</title>
      <table>
        <search>
          <query>
            index=os  sourcetype=df $host$ 
            | dedup host 
            | multikv 
            | table Filesystem Type Size Used Avail UsePct MountedOn
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">100</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Processes</title>
      <table>
        <search>
          <query>index=os sourcetype=ps $host$ 
            | multikv 
            | table USER PID PSR pctCPU CPUTIME pctMEM RSZ_KB VSZ_KB TTY S ELAPSED COMMAND ARGS</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>netstat</title>
      <table>
        <search>
          <query>index=os sourcetype=netstat $host$ 
            | dedup host 
            | multikv 
            | table Proto Recv-Q Send-Q LocalAddress ForeignAddress State</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
    <panel>
      <title>packages</title>
      <table>
        <search>
          <query>index=os sourcetype=package $host$ 
            | multikv 
            | dedup host NAME 
            | table NAME VERSION RELEASE ARCH VENDOR GROUP 
            | sort NAME</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>top command</title>
      <table>
        <search>
          <query>index=os sourcetype=top $host$ 
            | dedup host 
            | multikv 
            | table PID USER PR NI VIRT RES SHR S pctCPU pctMEM cpuTIME COMMAND</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>protocol</title>
      <table>
        <search>
          <query>index=os sourcetype=protocol $host$ 
            | dedup host 
            | multikv 
            | table IPdropped TCPrexmits TCPreorder TCPpktRecv TCPpktSent UDPpktLost UDPunkPort UDPpktRecv UDPpktSent</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>openPorts</title>
      <table>
        <search>
          <query>index=os sourcetype=openPorts $host$ 
            | dedup host 
            | multikv 
            | table Proto Port</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
    <panel>
      <title>Users with private logins</title>
      <table>
        <search>
          <query>index=os sourcetype=usersWithLoginPrivs $host$ 
            | dedup host 
            | multikv 
            | table USERNAME HOME_DIR USER_INFO</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">100</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="number" field="Floppy Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Hard Disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Virtual disk">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="RAM">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Swap">
          <option name="unit">GB</option>
        </format>
        <format type="number" field="Cache">
          <option name="unit">kB</option>
        </format>
      </table>
    </panel>
  </row>
</form>

That you can use to find your searches.

Ciao.

Giuseppe

[indeed_2000]

I try to use this dashboard but get this error:

Error on line 52: Unexpected close tag

after remote this line get another error:

Error on line -1: Rows can only contain visualization elements or panels, not both.

any idea?

[hishamjan]

Hi Guiseppe,

I tried copy-pasting the entire message that you had just sent (with necessary changes made) but it doesn't fetch me any results. The error says, invalid argument 'index=os'.

What am I doing wrong?

Thanks for the botheration. 

[gcusello]

Hi @hishamjan,

I stored my linux events in an index called "os", where do you store your Linux events?

Ciao.

Giuseppe

[hishamjan]

Hi Guiseppe,

I've encircled my indexes which stores my Linux events. Had to cancel out some client details, hope you don't mind.

I hope it gives you an idea of how should I be optimizing my search.

Regards

[gcusello]

Hi @hishamjan,

I don't like to have events in main index so I hint to modify your input to send all Linux events to os index, anyway, you have two choices:

• replace every index=os in the dashboard with index=os OR index=main,
• create an eventtype (called e.g. "linux") containing the search index=os OR index=main and then replace every index=os in the dashboard with eventtype=linux

I always prefer the second, even if it requires more effort.

Ciao.

Giuseppe

[hishamjan]

Hi Guiseppe,

I received these files as it is otherwise I would've never let indexing into my main. 

Anyhow, according to your advice, I should copy the dashboard that you used for Linux Servers monitoring and replace index=os with an eventtype called Linux that searches index=os AND index=main, right?

Regards

[gcusello]

Hi @hishamjan,

yes exactly: in general I hint to follow always this approach in dashboards, the are more flexible.

Ciao.

Giuseppe

[hishamjan]

hi @gcusello 

I was wondering, can we make use of the Splunk Stream App for monitoring UDP and TCP traffic instead of using sourcetype=protocol ?

If so, I'd very much like to know how is that more beneficial than just the use of sourcetype for monitoring protcols.

Regards,

Hisham

[gcusello]

Hi @hishamjan,

this is a different argument and I'm not an expert on Splunk Stream App, so, please, open a a new question about this!

Anyway Splunk Stream App is part of the purpose-built wire data collection and analytics solution from Splunk along with Splunk Add-on for Stream Forwarders for data collection and Splunk Add-on for Stream Wire Data for data parsing and formatting.

In few words it permits to perform Packet Capture and monitor traffic for security.

Ciao.

Giuseppe