Deployment Architecture

My goal is to find out statistics for particular error in all servers.

pwongcha
Explorer

My goal is to find out statistics for particular error in all servers.

Scenario
what host have error 'E' the most?
display in table/chart
group by time, host, sourcetype

Condition
1. Only 9am-5pm
2. Only M-F
3. Display in 3 hours bucket

Advanced Feature
1. Give me top 5 servers with the most errors
2. Include date_wday in the result

Base on the research, it looks like I cannot use timechart over 2 fields.

earliest=-90d host=cswebprd1* sourcetype=psoft* "java.lang.OutOfMemoryError" "Java heap space" NOT(date_wday="sunday") NOT(date_wday="saturday") (date_hour>= 9 AND date_hour<= 17)
| bucket _time span=6h
| stats count by _time date_wday host sourcetype
| table sourcetype count _time sourcetype date_wday host
| sort sourcetype

alt text
alt text

Tags (2)
0 Karma

tedwroks
Explorer

The interpretation is that per 3 hour bucket, you want to see the host source combination that had the most errors.
This is done with the last 3 lines that add a new field, "rank", and it's used to filter the result later.

    earliest=-90d    host=cswebprd1* sourcetype=psoft*
    java.lang.OutOfMemoryError    "Java heap space"
    NOT date_wday=sunday  NOT date_wday=saturday
    date_hour>= 9 date_hour<= 17
    | bucket _time span=3h
    | eval hostsrc=host.sourcetype
    | stats count as number_of_errors by _time, date_wday, hostsrc

    | sort 0 _time, -number_of_errors
    | streamstats count as rank by _time reset_on_change=true
    | where rank <= 3
0 Karma

woodcock
Esteemed Legend

You can use a trick to timechart over 2 fields like this:

... | eval comboField = field1 . ":" . field2 | timechart count by foo over comboField
0 Karma

pwongcha
Explorer

| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

with above command, I get this chart. Too many rows with 0 count. can this be removed from the result.

Looks like I get don't get 0 count from command "| bucket _time span=6h | stats count by _time date_wday host sourcetype"

0 Karma

woodcock
Esteemed Legend

Yes, when you use timechart it creates 0-value data points but when you use stats it does not do this. You can chart the output from stats if you go to the visualization tab.

0 Karma

pwongcha
Explorer

...
| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

almost there but I still prefer chart format done in excel.

is it possible to do this in plunk?

I can't attach image this time...

0 Karma

woodcock
Esteemed Legend

I do not understand this comment; please elaborate.

0 Karma

pwongcha
Explorer

I tried, doesn't look like timechart and over can be used together

0 Karma

woodcock
Esteemed Legend

You are correct (by the way, this is because it is already doing an implied over _time); I never should have used over but the basic trick still works like this:

... | eval comboField = field1 . ":" . field2 | timechart count by comboField

Also, this may help:

http://answers.splunk.com/answers/59045/how-do-i-make-a-multi-dimension-timechart.html

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!