How can I monitor network device or server on splunk ? like pingstatus command or something?

[modernjameschen]

Hello everyone,
I am a Rookie, I use splunk for linux,I tried running pingstatus command on splunk
But I don’t know if it was successful, I also read Readme.txt
and configured in commands.conf and authorize.conf
Can someone answer or teach me ? or there are other methods or applications that can monitor network devices
and if package have loss 100% then send a alert notification
It's best to start over
The more detailed the better
Thanks everyone

[woodcock]

Splunk is not really that kind of tool. You would be better of using a tool specifically designed for this purpose. I can highly recommend cacti:
https://en.wikipedia.org/wiki/Cacti_(software)

[modernjameschen]

HIi @woodcock
Thanks for your response
But my job needs to use splunk to monitor the device

[DavidHourani · Accepted solution]

Hi @modernjameschen,

Are you using this pingstatus command from Splunkbase ?
https://splunkbase.splunk.com/app/507/

If so I recommend moving to this as it is more up to date and supports more recent Splunk versions :
https://splunkbase.splunk.com/app/3491/
Once you install it you can find the documentation for using it here : https://lukemurphey.net/projects/network-tools/wiki/Using_Search_Commands
It's pretty easy to use and from there you can set up alerts to alert you when you have ping failures.

Let me know if that helps.

Cheers,
David

[modernjameschen]

Hi @DavidHourani ,
Thank you for your quick response
Yes,l using "pingstatus" command from Splunkbase
But I still don't understand how pingstatus works and configures
I want to know immediately the status of the ping ,Iike Destination host 、Feedback time、Packet loss rate ...
Or you suggest me to use NETWORK TOOLKIT
Which one is easier to get started
Or any idea
james,Thanks

[DavidHourani]

you're welcome @modernjameschen.
Both are easy to use but Network Toolkit is supported in the more recent splunk versions, whereas pingstatus is only supported up until 6.2 (you can see that on the right side when you're on the app url for splunkbase)

For network toolkit, if you need a tutorial, check out the link I sent you earlier : https://lukemurphey.net/projects/network-tools/wiki/Using_Search_Commands
In short, all you have to do is do a |ping on a table with host ip or domain names and youll get the ping results

[modernjameschen]

Thank @DavidHourani .
I started running ping in network toolkit
But I have a question, how can I make the setting return a Response Time every second
And how to set an alert when the ping packet is lost
Or need to configure anything else
Any ideas?

[DavidHourani]

For alerts you have to create an alert or a scheduled report that will run every X amount of time and do the ping for u.

As for modifying the default behavior of the ping command you'll have to modify the scripts in the bin folder of the app.

[modernjameschen]

@DavidHourani
I already have search
[sourcetype = "ping_input" packet_loss = 100]
Saved as an alert, but the alert is not working
Alarm type: real-time
Trigger: Number of sources equals 100
Once in 1 minute
Per-Result trigger option.

Trigger action: send email

And I don't know where to set the script
In splunk / etc / apps / network_tool / bin ?
Which folder?
Please help me

---

19/12/05 16:29:16.000

sent=3 received=0 packet_loss=100 min_ping=NA avg_ping=NA max_ping=NA jitter=NA return_code=1 dest=192.168.0.12
host = ubuntusource = pingsourcetype = ping_input

Above is my search: [sourcetype = "ping_input" packet_loss = 100]

[DavidHourani]

Hi @modernjameschen,

Is the alert created in the same app ? Could be that you don't have permissions to access the field extraction in another app.

Try changing the permissions of the network toolkit app to global and also include the index name in front of the sourcetype for the search :

index=yourindex sourcetype = "ping_input" packet_loss = 100

[modernjameschen]

Hi @DavidHourani
Yes ，I am created alert in the network toolkit app,
and I setup the permissions of the network toolkit app to global ，But I have a question, why should I create an index
Where to go to add a profile index
Which folder?
Can you give me steps
thank you very much

[DavidHourani]

Sure thing, to create an index follow those steps :
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Setupmultipleindexes#Create_events_indexe...

Your data is already written in an index by default somewhere (could be in index=main if you haven't configured anything). Run this search and check on the left side which index it's writting to, if it's main make sure you change it into something else :
sourcetype = "ping_input" packet_loss = 100

Let me know if that helps.

Cheers,
David

[modernjameschen]

hi @DavidHourani
I change my search

index = main sourcetype = "ping_input" packet_loss = 100

And also set alert
But the alert still not working
I don't know what went wrong
Is it possible to trigger a condition?
I don't know which one to choose
(Number of Results
Source of Results
Host of Results)

Please help me
thank you very much

---

Alarm type: real-time
Number of Results
Trigger: Number of sources equals 100
Once in 1 minute
Per-Result trigger option.
Trigger action: send email

[DavidHourani]

Try the default trigger... Number of results >0

This should trigger, alsp add as an action to trigger an alert in Splunk so u can actually see if anything happened. In case your send email function is not configured you will be able to see the alert and isolate the problem as a mailing problem

[modernjameschen]

hi @DavidHourani ,
I changed the trigger condition to
Number of results >0
Also change the trigger action to
Add to Triggered Alerts list and sent email.
I can see that Triggered Alerts have events.
But it still cann't to send email
It's very confused to me
Any ideal ?
Thanks ，james

[DavidHourani]

Could be that your instance isnt configured for sending emails ?

Check your general settings for the email configuration and make sure its set up:
https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/Alert/Emailnotification

[modernjameschen]

hi @DavidHourani

I have configured splunk instance
But I could n’t receive EMAIL, so I checked $ SPLUNK_HOME / var / log / splunk / splunkd.log
An error message appears

12-14-2019 10:10:00.256 +0800 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=http://user:8000/app/search/@go?sid=rt_scheduler__admin__search__RMD5f497f97cb39c3595_at_1576144552_1.21" "ssname=test 2f2p error" "graceful=True" "trigger_time=1576289398" results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD5f497f97cb39c3595_at_1576144552_1.21/results.csv.gz" "is_stream_malert=False"':  ERROR:root:{u'myemail@mdg.com.tw': (553, '5.7.1 <splunk@user>: Sender address rejected: not owned by user Username@mdg.com.tw')} while sending mail to: myemail@mdg.com.tw

Have any ideal?
Thanks a lot

[DavidHourani]

Seems like your sender email is not configured properly.. try changing it and see what it gives... Should be username@FQDN

[modernjameschen]

Thanks @DavidHourani
I changed the Mail host and Email securtiy
Then successfully send email

Mail host : smtp.gmail.com:587
Email security : Enable TLS

Can send mail with gmail account

[DavidHourani]

Awesome! please upvote my useful comments and accept the answer ! 🙂

[modernjameschen]

hi @DavidHourani

I met a problem
I try to create a monitoring dashboard
Save each search result as a monitoring dashboard
But i have a problem
You ca n’t add more than seven search results
I try to edit the source of the dashboard
Can increase search results but not perform
Can only perform up to seven searches
Below is my code
Any ideas

<dashboard>
  <label>Monitor device</label>
  <row>
    <panel>
      <title>2F-AP : 192.168.0.12</title>
      <single>
        <title>192.168.0.12 happne error</title>
        <search>
          <query>index = main "dest=192.168.0.12"  | chart  avg(packet_loss)</query>
          <earliest>rt-60s</earliest>
          <latest>rtnow</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>10m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorBy">value</option>
        <option name="colorMode">block</option>
        <option name="drilldown">none</option>
        <option name="numberPrecision">0.0</option>
        <option name="rangeColors">["0x53a051","0x53a051","0xf8be34","0xf1813f","0xdc4e41","0xdc4e41"]</option>
        <option name="rangeValues">[0,26,51,76,100]</option>
        <option name="refresh.display">progressbar</option>
        <option name="showSparkline">1</option>
        <option name="showTrendIndicator">1</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
        <option name="trendColorInterpretation">standard</option>
        <option name="trendDisplayMode">absolute</option>
        <option name="unitPosition">after</option>
        <option name="useColors">1</option>
        <option name="useThousandSeparators">1</option>
      </single>
    </panel>
．
．
．
 </row>
</dashboard>

Or where I can setup limit
Thanks a lot