Splunk Search

How to document a Splunk query

sonam
Explorer

I am writing a Splunk query to search logs generated by a middleware system for anything 'exceptional'. Basically, the approach I'm taking is filtering out entries for 'known issues'. Anything left behind is then an 'unknown issue' by definition (and worthy of attention). The Splunk query examines the previous day's logs each night and emails results for review in the morning.

The query looks like this... as you can see, it just a large set of 'NOT' terms:

index=middleware
NOT SalesForce* 
NOT SSL_DEBUG
NOT "Cache cleared for service *" 
NOT "Service Thread Pool" 
...
(20 more exclusions and growing)
...

My questions :

  1. How can I document this query?
    Specifically, I'd like to explain each 'NOT' exclusion above.

  2. Is this a reasonable approach for reviewing logfiles?
    My Splunk admin is concerned about the performance impact of 'NOT' terms.

jbsplunk
Splunk Employee
Splunk Employee

You could add it into savedsearches.conf, and then just add the comments in above that, specifying they are comments by beginning each comment line with a # symbol. In terms of a query expense, your returning the entire set of results, and then filtering based on that. It is better if you can specify a time frame, and the results that you'd like to see being as specific as possible prior to filtering out events.

0 Karma

sonam
Explorer

My impression is there is no functionality available to a Splunk end-user, to document Splunk artifacts in Splunk.

The only option seems to be to copy/paste the saved searches/events, etc... from Splunk into a Wiki or Word files or whatever knowledgebase you use, and document it there.

0 Karma

sonam
Explorer

Hmm. Thanks for that. You mentioned a configuration file (savedsearches.conf). However, I'm just a poor user, not a Splunk sysadmin so I don't have access to this file. (Am I wrong?) I can only save searches and event.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...