Archive

How can we indetify indexes which get currently lots of DEBUG events?

Ultra Champion

We got a license warning yesterday and we are pretty sure it's due to excessive DEBUG events coming through. Is it possible to create a report specifying the top current indexes with DEBUG events?

Is there a way to intercept the DEBUG events at parsing time and discard them?

Tags (2)
0 Karma
1 Solution

Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

View solution in original post

0 Karma

Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

View solution in original post

0 Karma

Ultra Champion

Great. Normally the events contain the word DEBUG in upper-case. Just based on that, can we create a query which would give a break-up of today's DEBUG data by the indexes?

index=* DEBUG | stats count by index | sort - count is not bad - how can I enforce only upper-case DEBUG?

0 Karma

Splunk Employee
Splunk Employee

@somesoni2 is right this search will be a heavy hit in terms of performance so make sure to filter by time so that you are only searching what you have not already checked. Also, I would bet that this only occurs in specific indexes and sourcetypes. Filter by just those indexes and sourcetypes as well.

0 Karma

Ultra Champion

Right right Claw - scary to run it in production, which I'm doing now ; - )

0 Karma

Revered Legend

Well, it'll be long *** query.

index=* CASE(DEBUG) | stats count by index

or useful but even worst in terms of performance.

index=* | eval isDebug=if(searchmatch("DEBUG"),1,0) | stats count as Total sum(isDebug) as Debug by index | eval Perc=Debug*100/Total
0 Karma

Ultra Champion

Wow - gorgeous

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!