Deployment Architecture

How can we indetify indexes which get currently lots of DEBUG events?

ddrillic
Ultra Champion

We got a license warning yesterday and we are pretty sure it's due to excessive DEBUG events coming through. Is it possible to create a report specifying the top current indexes with DEBUG events?

Is there a way to intercept the DEBUG events at parsing time and discard them?

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

View solution in original post

0 Karma

somesoni2
Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

0 Karma

ddrillic
Ultra Champion

Great. Normally the events contain the word DEBUG in upper-case. Just based on that, can we create a query which would give a break-up of today's DEBUG data by the indexes?

index=* DEBUG | stats count by index | sort - count is not bad - how can I enforce only upper-case DEBUG?

0 Karma

Claw
Splunk Employee
Splunk Employee

@somesoni2 is right this search will be a heavy hit in terms of performance so make sure to filter by time so that you are only searching what you have not already checked. Also, I would bet that this only occurs in specific indexes and sourcetypes. Filter by just those indexes and sourcetypes as well.

0 Karma

ddrillic
Ultra Champion

Right right Claw - scary to run it in production, which I'm doing now ; - )

0 Karma

somesoni2
Revered Legend

Well, it'll be long *** query.

index=* CASE(DEBUG) | stats count by index

or useful but even worst in terms of performance.

index=* | eval isDebug=if(searchmatch("DEBUG"),1,0) | stats count as Total sum(isDebug) as Debug by index | eval Perc=Debug*100/Total
0 Karma

ddrillic
Ultra Champion

Wow - gorgeous

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...