topic Re: Send events to null queue which dont have time in Getting Data In

Send events to null queue which dont have time

power12 — Sat, 24 Feb 2024 00:01:08 GMT

I have the following sample events coming from
source="/project/admin/git/ys/es/perf/de/pure/abc0*/logs/*/results.csv"

Event1: with no timestamp and these type of data is in files which are 2 days older

abc|pxyz|0.1054|ops|0|null|null

Event 2 with timestamp --these are new files from same location and going forward the data would be the same as below

2024-02-23T00:48:17|AID|read|454482.351348|PS|0|null|null

I want to send data to splunk that has a timestamp and send the other to null queue or not to ingest it .

Firstly I tried MAX_DAYS_AGO=2 which did not work then I tried the following props and transforms but it did not work .

transforms [filter] REGEX = ^^\D* DEST_KEY = queue FORMAT = nullQueue props.conf CHARSET=AUTO SHOULD_LINEMERGE=false category=Custom disabled=false pulldown_type=true TRANSFORMS-null=filter

Thanks in Advance

Re: Send events to null queue which dont have time

richgalloway — Sat, 24 Feb 2024 01:13:49 GMT

The regex is filtering lines beginning with "^", which aren't in your sample data. Assuming that's a mistake, it's also filtering on any number of non-digits (including zero), which pretty much matches anything. Try these settings

transforms [filter] REGEX = ^\D+ DEST_KEY = queue FORMAT = nullQueue props.conf TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%dT%H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 20 CHARSET=AUTO SHOULD_LINEMERGE=false LINE_BREAKER = ([\r\n]+) category=Custom disabled=false pulldown_type=true TRANSFORMS-null=filter

Re: Send events to null queue which dont have time

PickleRick — Sat, 24 Feb 2024 07:42:33 GMT

You could also go the other way around. Do the nullQueue by default and only send to indexQueue those that _do_ match the timestamp regex.