How can we indetify indexes which get currently lots of DEBUG events?

[ddrillic]

We got a license warning yesterday and we are pretty sure it's due to excessive DEBUG events coming through. Is it possible to create a report specifying the top current indexes with DEBUG events?

Is there a way to intercept the DEBUG events at parsing time and discard them?

[somesoni2 · Accepted solution]

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

[ddrillic]

Great. Normally the events contain the word DEBUG in upper-case. Just based on that, can we create a query which would give a break-up of today's DEBUG data by the indexes?

index=* DEBUG | stats count by index | sort - count is not bad - how can I enforce only upper-case DEBUG?

[Claw]

@somesoni2 is right this search will be a heavy hit in terms of performance so make sure to filter by time so that you are only searching what you have not already checked. Also, I would bet that this only occurs in specific indexes and sourcetypes. Filter by just those indexes and sourcetypes as well.

[ddrillic]

Right right Claw - scary to run it in production, which I'm doing now ; - )

[somesoni2]

Well, it'll be long *** query.

index=* CASE(DEBUG) | stats count by index

or useful but even worst in terms of performance.

index=* | eval isDebug=if(searchmatch("DEBUG"),1,0) | stats count as Total sum(isDebug) as Debug by index | eval Perc=Debug*100/Total

[ddrillic]

Wow - gorgeous