How to route specific index data regardless of any source to a null queue?

[arunsunny]

For example:
I have more than 1000 source data coming with a different more than 1000 sourcetype into a specific index called "cricket".

Now I have to route all these data to a null queue.

Currently, I'm trying below props.conf and transforms.conf but not working

props.conf
[source::*]
TRANSFORMS-null= setnull

transforms.conf
[setnull]
SOURCE_KEY = _MetaData:cricket
DEST_KEY = queue
FORMAT = nullQueue

Any suggestions/inputs ?

Cheers,
Arun Sunny

[arunsunny · Accepted solution]

Hi @gcusello and @martynoconnor ,

I solved my problem by using the below props and transforms and appreciate your help 🙂

props.conf
[source::...]
TRANSFORMS-null= setnullCricket

transforms.conf
[setnullCricket]
SOURCE_KEY = _MetaData:Index
REGEX = (cricket)
DEST_KEY = queue
FORMAT = nullQueue

[martynoconnor]

props.conf:

[default]
TRANSFORMS-killthecrickets

transforms.conf:

[killthecrickets]
REGEX=cricket
SOURCE_KEY=_Metadata:index
DEST_KEY=queue
FORMAT=nullQueue

[gcusello]

Hi arunsunny,
let me understand: why do you ingest this logs if you want to discard them?
maybe it's easier to exclude these files from input using blacklists.
In this way you have also less bandwidth occupation and work for the Forwarders.

Or, if you cannot exclude them from input, give them a unic sourcetype (or sourcetypes that start with a specified prefix) so it's easy to discard them.

Bye.
Giuseppe

[arunsunny]

Thanks, @gcusello for the quick response.

The problem which I have is we don't have any control over the inputs. ( The end users can send data with any source/sourcetype, So I need remediate that on our end)

My problem statement is - I need to discard the events regardless of any source/sourcetype coming to a specific index.

[gcusello]

Hi arunsunny,
this isn't a good idea because you haven't any control on license consuption and bandwidth occupation!

Anyway, if the license isn't a problem for you, the easiest way to proceed is to set a very little retention for data on cricket index (e.g. 3600 seconds) in this way the logs in this index are deleted in a little time:
to do this, insert the option frozenTimePeriodInSecs = 3600 in indexes.conf file, cricket stanza.

If you don't want to index these logs, you should try something like this:

[setnull]
SOURCE_KEY = field:index
REGEX = cricket
DEST_KEY = queue
FORMAT = nullQueue

I'm not sure of this because I didn't do or see something like you asked!

Bye.
Giuseppe