I have more than 1000 source data coming with a different more than 1000 sourcetype into a specific index called "cricket".
Now I have to route all these data to a null queue.
Currently, I'm trying below props.conf and transforms.conf but not working
SOURCE_KEY = _MetaData:cricket
DEST_KEY = queue
FORMAT = nullQueue
Any suggestions/inputs ?
let me understand: why do you ingest this logs if you want to discard them?
maybe it's easier to exclude these files from input using blacklists.
In this way you have also less bandwidth occupation and work for the Forwarders.
Or, if you cannot exclude them from input, give them a unic sourcetype (or sourcetypes that start with a specified prefix) so it's easy to discard them.
Thanks, @gcusello for the quick response.
The problem which I have is we don't have any control over the inputs. ( The end users can send data with any source/sourcetype, So I need remediate that on our end)
My problem statement is - I need to discard the events regardless of any source/sourcetype coming to a specific index.
this isn't a good idea because you haven't any control on license consuption and bandwidth occupation!
Anyway, if the license isn't a problem for you, the easiest way to proceed is to set a very little retention for data on cricket index (e.g. 3600 seconds) in this way the logs in this index are deleted in a little time:
to do this, insert the option
frozenTimePeriodInSecs = 3600 in indexes.conf file, cricket stanza.
If you don't want to index these logs, you should try something like this:
[setnull] SOURCE_KEY = field:index REGEX = cricket DEST_KEY = queue FORMAT = nullQueue
I'm not sure of this because I didn't do or see something like you asked!