Searching across multiple saved searches

[bjwarner]

Hi there,

I am trying to use splunk to understand the alerts that are coming out of our system. We get approx 35K alerts per week. We currently use a series of bash/grep/awk scripts to process and get stats on these alerts, but I'm trying to see if it's any easier with splunk.

I've setup a number of saved searches in splunk which correspond to "known errors/warnings" which occur in our system. These are just text patterns. There will be approx 100 of these saved searches once I've finished entering them.

I'd like to build a report/chart, that we can run daily/weekly which shows:

1. The number of matches for each saved search across a given period.

2. Possibly some way of seeing a stacked histogram of matches for each saved search. This would aid in seeing correlations between different known errors.

3. The number of lines which do NOT match ANY saved search. This is valuable information as it points to errors/warnings that we don't know about.

I realize that this might be a big ask!

Cheers,
Ben

[emiller42 · Accepted solution]

This sounds like an excellent case for event types. (Event type Documentation)

Since your saved searches simply return events that match those criteria, you can use those searches as the basis for event type categorization. For example, if you use the following search as the basis for the event type 'tomcat_error'...

sourcetype=tomcat logLevel="ERROR"

Then any event that matches that criteria will automatically be flagged as eventtype=tomcat_error and you can then search for:

eventtype="tomcat_error"

and get all events that match the original search criteria.

Then instead of a hundred saved searches, you just define all of those as event types, and they become very easy to aggregate.

* | timechart count by eventtype

You can continue to flag new event types as you discover them, and the reports won't need to be changed as they're just working with that field.

[emiller42]

Whoops, that's right Lucas. Updating my answer. Thanks!

[Lucas_K]

Use the command "by" in your timechart command to separate your series. ie. timechart count by eventtype etc.

[bjwarner]

Hi there. Thanks very much for your answer. Event types look to be just what I'm after. The query you entered "* | timechart count(eventtype)" shows a total across all event types. Do you know how I would write a query to show each eventtype as a separate line on the graph?