Deployment Architecture

How can we indetify indexes which get currently lots of DEBUG events?

ddrillic
Ultra Champion

We got a license warning yesterday and we are pretty sure it's due to excessive DEBUG events coming through. Is it possible to create a report specifying the top current indexes with DEBUG events?

Is there a way to intercept the DEBUG events at parsing time and discard them?

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

View solution in original post

0 Karma

somesoni2
Revered Legend

To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.

0 Karma

ddrillic
Ultra Champion

Great. Normally the events contain the word DEBUG in upper-case. Just based on that, can we create a query which would give a break-up of today's DEBUG data by the indexes?

index=* DEBUG | stats count by index | sort - count is not bad - how can I enforce only upper-case DEBUG?

0 Karma

Claw
Splunk Employee
Splunk Employee

@somesoni2 is right this search will be a heavy hit in terms of performance so make sure to filter by time so that you are only searching what you have not already checked. Also, I would bet that this only occurs in specific indexes and sourcetypes. Filter by just those indexes and sourcetypes as well.

0 Karma

ddrillic
Ultra Champion

Right right Claw - scary to run it in production, which I'm doing now ; - )

0 Karma

somesoni2
Revered Legend

Well, it'll be long *** query.

index=* CASE(DEBUG) | stats count by index

or useful but even worst in terms of performance.

index=* | eval isDebug=if(searchmatch("DEBUG"),1,0) | stats count as Total sum(isDebug) as Debug by index | eval Perc=Debug*100/Total
0 Karma

ddrillic
Ultra Champion

Wow - gorgeous

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...