Dashboards & Visualizations

Monthly Dashboard

mvasquez21
Explorer

so here is my new assignment: we have about 15 alerts that search for various key phrases in our Cisco network's syslogs. Some of them return hundreds or even thousands of results (usually in a "|stats count" format). My boss wants me to create a dashboard with a time picker to see the daily number of returned results for each alerts in a stacked bar chart. So basically the X axis would be the date and the y-axis would be count with each alert represented by a different color. I am thinking i need to specify the search for each alert, then somehow assign it a variable to be specified in an "xyseries" command. I already have the date conversion from another dashboard:   convert timeformat="%m/%d" ctime(_time) AS date | stats count by "xyz",date | xyseries date, "xyz", count

What i can't seem to get is each alert's search string as a separate item in the series. Here is an example alert:

index=network key_word=*HWPORTMAN-*QUEUE OR key_word=LINECARDMGMTPROTOCOL-*WARNING | stats count by host

Labels (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could try evaluating an alert type with a case statement

index=network
| eval alert=case((key_word=*HWPORTMAN-*QUEUE OR key_word=LINECARDMGMTPROTOCOL-*WARNING), "1", (key_word=XYZ), "2", ...
| bin span=1d _time
| stats count by alert, _time

 

0 Karma

mvasquez21
Explorer

so im a little light on deep Splunk commands. This one is returning an error. Suppose I want to simplify and just use 1 string to test. 

index=network | eval alert=case (key_word=*HWPORTMAN-*QUEUE OR key_word=LINECARDMGMTPROTOCOL-*WARNING) | bin span=1d _time | stats count by alert, _time

This is still returning an error:

Error in 'eval' command: The expression is malformed. An unexpected character is reached at '*HWPORTMAN-*QUEUE OR key_word=LINECARDMGMTPROTOCOL-*WARNING)

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try putting the values in double quotes

index=network | eval alert=case (key_word="*HWPORTMAN-*QUEUE" OR key_word="LINECARDMGMTPROTOCOL-*WARNING") | bin span=1d _time | stats count by alert, _time
0 Karma

mvasquez21
Explorer

still get an error:

Error in 'eval' command: The arguments to the 'case' function are invalid.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Sorry, I missed that you had dropped the "1"

index=network | eval alert=case (key_word="*HWPORTMAN-*QUEUE" OR key_word="LINECARDMGMTPROTOCOL-*WARNING", "Alert1") | bin span=1d _time | stats count by alert, _time
0 Karma

mvasquez21
Explorer

OK this one seems to work. So what you're saying is i would need to replicate the "eval" section for each alert's search string  correct? (of course changing the "alert1" at the end to whatever the alert name is)

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Yes, the case function takes pairs of parameters, a condition and a value e.g case(condition1,value1,condition2,value2,...) and so on. This means the eval will set the field alert to whatever string value you choose to use for that condition. Then your stats can use this alert value to group your stats

Having said that. you should still look into summary indexes as @gcusello suggested (assuming you have permissions to do so), and you could run this query on a nightly basis or have lots of separate queries feeding the same summary index with a field to identify which alert condition the entry in the index refers to

0 Karma

mvasquez21
Explorer

Thanks but actually is seems the "case" parameter does not work as it does not perform the search in the parenthesis. It is returning every single syslog entry which is taking very long to complete. a 15 minute windows returns a count of over 14,000 when i know the search itself only has 1 hit in the last 24 hours.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could limit the events returned

index=network key_word="*HWPORTMAN-*QUEUE" OR key_word="LINECARDMGMTPROTOCOL-*WARNING" | eval alert=case (key_word="*HWPORTMAN-*QUEUE" OR key_word="LINECARDMGMTPROTOCOL-*WARNING", "Alert1") | bin span=1d _time | stats count by alert, _time

For multiple alert types, you would limit the search to all the possible alert conditions  

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mvasquez21,

a monthly report could be a very long search, in my opinion the best approach is to schedule a search every night that report all the fired alerts with the timestamp, saving results in a Summary Index, then you can search on the summary index and have a very quick result also with periods of months.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...