Getting Data In

How to route specific index data regardless of any source to a null queue?

arunsunny
Path Finder

For example:
I have more than 1000 source data coming with a different more than 1000 sourcetype into a specific index called "cricket".

Now I have to route all these data to a null queue.

Currently, I'm trying below props.conf and transforms.conf but not working

props.conf
[source::*]
TRANSFORMS-null= setnull

transforms.conf
[setnull]
SOURCE_KEY = _MetaData:cricket
DEST_KEY = queue
FORMAT = nullQueue

Any suggestions/inputs ?

Cheers,
Arun Sunny

0 Karma
1 Solution

arunsunny
Path Finder

Hi @gcusello and @martynoconnor ,

I solved my problem by using the below props and transforms and appreciate your help 🙂

props.conf
[source::...]
TRANSFORMS-null= setnullCricket

transforms.conf
[setnullCricket]
SOURCE_KEY = _MetaData:Index
REGEX = (cricket)
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

arunsunny
Path Finder

Hi @gcusello and @martynoconnor ,

I solved my problem by using the below props and transforms and appreciate your help 🙂

props.conf
[source::...]
TRANSFORMS-null= setnullCricket

transforms.conf
[setnullCricket]
SOURCE_KEY = _MetaData:Index
REGEX = (cricket)
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

martynoconnor
Communicator

props.conf:

[default]
TRANSFORMS-killthecrickets

transforms.conf:

[killthecrickets]
REGEX=cricket
SOURCE_KEY=_Metadata:index
DEST_KEY=queue
FORMAT=nullQueue

0 Karma

gcusello
Legend

Hi arunsunny,
let me understand: why do you ingest this logs if you want to discard them?
maybe it's easier to exclude these files from input using blacklists.
In this way you have also less bandwidth occupation and work for the Forwarders.

Or, if you cannot exclude them from input, give them a unic sourcetype (or sourcetypes that start with a specified prefix) so it's easy to discard them.

Bye.
Giuseppe

0 Karma

arunsunny
Path Finder

Thanks, @gcusello for the quick response.

The problem which I have is we don't have any control over the inputs. ( The end users can send data with any source/sourcetype, So I need remediate that on our end)

My problem statement is - I need to discard the events regardless of any source/sourcetype coming to a specific index.

0 Karma

gcusello
Legend

Hi arunsunny,
this isn't a good idea because you haven't any control on license consuption and bandwidth occupation!

Anyway, if the license isn't a problem for you, the easiest way to proceed is to set a very little retention for data on cricket index (e.g. 3600 seconds) in this way the logs in this index are deleted in a little time:
to do this, insert the option frozenTimePeriodInSecs = 3600 in indexes.conf file, cricket stanza.

If you don't want to index these logs, you should try something like this:

[setnull]
SOURCE_KEY = field:index
REGEX = cricket
DEST_KEY = queue
FORMAT = nullQueue

I'm not sure of this because I didn't do or see something like you asked!

Bye.
Giuseppe

0 Karma