topic Re: Automatically reject data when timestamp could not be assigned in Getting Data In

Automatically reject data when timestamp could not be assigned

guilmxm — Sat, 27 Sep 2014 11:14:50 GMT

Hi All,

Does anyone knows a way to automatically reject data when Splunk could not identify event timestamp ?

My goal is to radically prevent inconsistent data to be indexed, if the timestamp could not be identified then this should be considered as an anomaly and the data would be sent to nullqueue (for example) instead of being indexed.

Is that possible ?

Thanks for your help

Guilhem

Re: Automatically reject data when timestamp could not be assigned

yannK — Mon, 28 Sep 2020 17:43:20 GMT

You can create 2 transforms to apply to your data in this order :

the first one drop all the events to nullQueue
the second look for a regex matching a valid timestamp pattern, and send them back to the indexQueue.

Read this guide "Keep_specific_events_and_discard_the_rest"
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest,You can add a nullQueue filter that looks for a particular pattern of timestamp, and drop the events it's missing one.
PS : it will happen after the linebreaking and event merging.

Re: Automatically reject data when timestamp could not be assigned

guilmxm — Sun, 28 Sep 2014 08:20:38 GMT

Hi Yann,
Thanks, that's in deed the way to proceed