Splunk Search

How to run a diff search with a Head 2 command across multiple systems?

aferone
Builder

I have developed a search, with help years ago, that will show differences in a netstat command using "diff" and "head 2".

index=foo host=bar sourcetype=netstat 
| head 2 
| diff
| search NOT "Results are the Same"

The netstat runs every hour with a scripted input, and the search runs hourly to see if anything has changed. If it has, an alert fires.

As you can see, the search specifies a host. However, what if I wanted to run this search across many hosts? Would I have to create a separate search for each host? Or is there some Splunk magic I can utilize?

Thanks for your time!

0 Karma

somesoni2
Revered Legend

The diff command can only compare 2 result/row, so having multiple host entries would not work. If you don't need "diff command type output" and just looking to alert when current netstat output of a host is different then previous entry, you can do something like this.

index=foo sourcetype=netstat 
| table host _raw | dedup 2 host
| streamstats count as sno by host
| chart values(_raw) over host by sno
| where '1'!='2'

The dedup command will just list two entries for a host, most recent and 2nd recent. The streamstats command just give a serial number to them which'll be 1 and 2 since there will be only two entries after dedup. The chart command will give a output with field host, 1 (which will have most recent event's raw data) and 2 (which will have 2nd recent event's raw data). The where clause just compare both.

0 Karma

aferone
Builder

This looks very promising. I am pushing our netstat config to more boxes to test this. Where does "sno" come into play in the search? Thanks again!!

0 Karma

aferone
Builder

The search is definitely working, and thank you!

But for some reason, when I set it up as an alert, I can't get it to send an email? I am setting it to "events greater to zero", just like every other alert we've configured.

Am I missing something?

Thanks!

0 Karma

aferone
Builder

Sorry for the delay, and thank you for posting! . I will be trying this soon. Thanks again!

0 Karma

pradeepkumarg
Influencer

Can you not do dedup host?


index=foo host=* sourcetype=netstat
| dedup 2 host

0 Karma

aferone
Builder

Can you explain this a little further on what this would be doing? Thanks!

0 Karma

pradeepkumarg
Influencer

so | head 2 will give you the latest 2 entries for the host you specified.

by making host=* and adding | dedup 2 host, you are retaining the latest 2 entries for every host

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...