Deployment Architecture

My goal is to find out statistics for particular error in all servers.

pwongcha
Explorer

My goal is to find out statistics for particular error in all servers.

Scenario
what host have error 'E' the most?
display in table/chart
group by time, host, sourcetype

Condition
1. Only 9am-5pm
2. Only M-F
3. Display in 3 hours bucket

Advanced Feature
1. Give me top 5 servers with the most errors
2. Include date_wday in the result

Base on the research, it looks like I cannot use timechart over 2 fields.

earliest=-90d host=cswebprd1* sourcetype=psoft* "java.lang.OutOfMemoryError" "Java heap space" NOT(date_wday="sunday") NOT(date_wday="saturday") (date_hour>= 9 AND date_hour<= 17)
| bucket _time span=6h
| stats count by _time date_wday host sourcetype
| table sourcetype count _time sourcetype date_wday host
| sort sourcetype

alt text
alt text

Tags (2)
0 Karma

tedwroks
Explorer

The interpretation is that per 3 hour bucket, you want to see the host source combination that had the most errors.
This is done with the last 3 lines that add a new field, "rank", and it's used to filter the result later.

    earliest=-90d    host=cswebprd1* sourcetype=psoft*
    java.lang.OutOfMemoryError    "Java heap space"
    NOT date_wday=sunday  NOT date_wday=saturday
    date_hour>= 9 date_hour<= 17
    | bucket _time span=3h
    | eval hostsrc=host.sourcetype
    | stats count as number_of_errors by _time, date_wday, hostsrc

    | sort 0 _time, -number_of_errors
    | streamstats count as rank by _time reset_on_change=true
    | where rank <= 3
0 Karma

woodcock
Esteemed Legend

You can use a trick to timechart over 2 fields like this:

... | eval comboField = field1 . ":" . field2 | timechart count by foo over comboField
0 Karma

pwongcha
Explorer

| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

with above command, I get this chart. Too many rows with 0 count. can this be removed from the result.

Looks like I get don't get 0 count from command "| bucket _time span=6h | stats count by _time date_wday host sourcetype"

0 Karma

woodcock
Esteemed Legend

Yes, when you use timechart it creates 0-value data points but when you use stats it does not do this. You can chart the output from stats if you go to the visualization tab.

0 Karma

pwongcha
Explorer

...
| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

almost there but I still prefer chart format done in excel.

is it possible to do this in plunk?

I can't attach image this time...

0 Karma

woodcock
Esteemed Legend

I do not understand this comment; please elaborate.

0 Karma

pwongcha
Explorer

I tried, doesn't look like timechart and over can be used together

0 Karma

woodcock
Esteemed Legend

You are correct (by the way, this is because it is already doing an implied over _time); I never should have used over but the basic trick still works like this:

... | eval comboField = field1 . ":" . field2 | timechart count by comboField

Also, this may help:

http://answers.splunk.com/answers/59045/how-do-i-make-a-multi-dimension-timechart.html

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...