I have multiple servers for which I am monitoring event logs via Splunk. The servers are owned by different teams. There is no information about team in the event log messages. I want to group the servers via team names in "one" graph (dashboard or report). The mapping between team and servers is internal.
Team A = Server1, Server 3, Server 5
Team B = Server2, Server6
Team C = Server4, Server7, Server8
Event logs have Host field holding server name (e.g: Server3). But no information about team is stored in the event log.
I want one panel which will show errors in last 24 hours by team.
X-Axis: Timespan count by hour
Y-Axis: Number of errors
3 columns per hour - one for each team
Query for errors by host:
(Type="Error") (wmi_type="WinEventLog:Application") | timechart span=1h count by host
Since field for team does not exist, I cannot use avg.
I tried to use subsearch with but it was giving fewer results than what I could get from the above query which tells me it is not correct.
How do I query the report?
you could use eventtype to group the servers into teams, take a look at the docs. So you can build an eventtype named
TeamA which referees to a search like this
host=Server1 OR host=Server OR host=Server 5
Next you can use this
eventtype in the search like this:
eventtype=TeamA (Type="Error") (wmi_type="WinEventLog:Application") | timechart span=1h count by host
hope this helps to get you started ...
I think this is exactly what I am looking for. I will work on it and keep this thread updated.
Update: This is it 🙂