Getting Data In

How to drop all entries to a specific index?

mlindsey
Explorer

We've reached our license limit. So, at the indexer, I want to drop all log entries destined to a specific index. Documentation is clear how to do that on a heavy forwarder, for example, but I haven't found any documentation of how to drop all traffic to a specific index at the indexer. Props.conf looked promising but it doesn't support an index key. In props.conf, I was expecting that I could create a stanza like this:

[index::development] # This key is not listed in the props.conf.spec

TRANSFORMS-blackhole = blackhole

and in transforms.conf:

[blackhole]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

It just seems there has to be a way, but I haven't been able to discover it.

0 Karma
1 Solution

mlindsey
Explorer

The following worked just great! Kudos to James Brodsky and Splunk Support.

In props.conf:

# This stanza drops alls events destined for the 'development' index.

#

[host::*]

TRANSFORMS-dropindex = senddevidxtonull

In transforms.conf:

[senddevidxtonull]

SOURCE_KEY=_MetaData:Index

REGEX=development

DEST_KEY=queue

FORMAT=nullQueue

View solution in original post

0 Karma

mlindsey
Explorer

The following worked just great! Kudos to James Brodsky and Splunk Support.

In props.conf:

# This stanza drops alls events destined for the 'development' index.

#

[host::*]

TRANSFORMS-dropindex = senddevidxtonull

In transforms.conf:

[senddevidxtonull]

SOURCE_KEY=_MetaData:Index

REGEX=development

DEST_KEY=queue

FORMAT=nullQueue

0 Karma

somesoni2
Revered Legend

As far as I know, the filtering can be done at source, sourcetype or host level, its not possible for index level. I would be interested to see if there really is a way.

0 Karma

jhupka
Path Finder

Wouldn't your REGEX=. just match single character events for sending to nullQueue? Maybe try REGEX=.* to make it more greedy and match everything.

0 Karma

jhupka
Path Finder

So I don't really have a quick and dirty answer to shutting off an index that doesn't come with its own caveats. E.g. if I had to do this I might try something like:

  • Just manually off the inputs going to that index and deploy it out real quick using DS.

  • If you don't care about anything in that index for now, you could just remove its config from your index.conf. But you'll get warnings when stuff goes to that index, but those events get dropped and I'm pretty sure don't affect license. Just don't delete the index's directories so when you re-enable the index old data is still there

0 Karma

mlindsey
Explorer

The REGEX is really irrelevant in this case because the example is a fantasy not an actual configuration. I just hoped it would be that way. The real issue is routing all traffic destined to one index to the nullQueue.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...