My goal is to find out statistics for particular error in all servers.

[pwongcha]

My goal is to find out statistics for particular error in all servers.

Scenario
what host have error 'E' the most?
display in table/chart
group by time, host, sourcetype

Condition
1. Only 9am-5pm
2. Only M-F
3. Display in 3 hours bucket

Advanced Feature
1. Give me top 5 servers with the most errors
2. Include date_wday in the result

Base on the research, it looks like I cannot use timechart over 2 fields.

earliest=-90d host=cswebprd1* sourcetype=psoft* "java.lang.OutOfMemoryError" "Java heap space" NOT(date_wday="sunday") NOT(date_wday="saturday") (date_hour>= 9 AND date_hour<= 17)
| bucket _time span=6h
| stats count by _time date_wday host sourcetype
| table sourcetype count _time sourcetype date_wday host
| sort sourcetype

[tedwroks]

The interpretation is that per 3 hour bucket, you want to see the host source combination that had the most errors.
This is done with the last 3 lines that add a new field, "rank", and it's used to filter the result later.

    earliest=-90d    host=cswebprd1* sourcetype=psoft*
    java.lang.OutOfMemoryError    "Java heap space"
    NOT date_wday=sunday  NOT date_wday=saturday
    date_hour>= 9 date_hour<= 17
    | bucket _time span=3h
    | eval hostsrc=host.sourcetype
    | stats count as number_of_errors by _time, date_wday, hostsrc

    | sort 0 _time, -number_of_errors
    | streamstats count as rank by _time reset_on_change=true
    | where rank <= 3

[woodcock]

You can use a trick to timechart over 2 fields like this:

... | eval comboField = field1 . ":" . field2 | timechart count by foo over comboField

[pwongcha]

| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

with above command, I get this chart. Too many rows with 0 count. can this be removed from the result.

Looks like I get don't get 0 count from command "| bucket _time span=6h | stats count by _time date_wday host sourcetype"

[woodcock]

Yes, when you use timechart it creates 0-value data points but when you use stats it does not do this. You can chart the output from stats if you go to the visualization tab.

[pwongcha]

...
| eval comboField= sourcetype.":".host
| timechart span=6h count by comboField

almost there but I still prefer chart format done in excel.

is it possible to do this in plunk?

I can't attach image this time...

[woodcock]

I do not understand this comment; please elaborate.

[pwongcha]

I tried, doesn't look like timechart and over can be used together

[woodcock]

You are correct (by the way, this is because it is already doing an implied over _time); I never should have used over but the basic trick still works like this:

... | eval comboField = field1 . ":" . field2 | timechart count by comboField

Also, this may help:

http://answers.splunk.com/answers/59045/how-do-i-make-a-multi-dimension-timechart.html