I have the following sample events coming from
source="/project/admin/git/ys/es/perf/de/pure/abc0*/logs/*/results.csv"
Event1: with no timestamp and these type of data is in files which are 2 days older
abc|pxyz|0.1054|ops|0|null|null
Event 2 with timestamp --these are new files from same location and going forward the data would be the same as below
2024-02-23T00:48:17|AID|read|454482.351348|PS|0|null|null
I want to send data to splunk that has a timestamp and send the other to null queue or not to ingest it .
Firstly I tried MAX_DAYS_AGO=2 which did not work then I tried the following props and transforms but it did not work .
transforms
[filter]
REGEX = ^^\D*
DEST_KEY = queue
FORMAT = nullQueue
props.conf
CHARSET=AUTO
SHOULD_LINEMERGE=false
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-null=filter
Thanks in Advance
The regex is filtering lines beginning with "^", which aren't in your sample data. Assuming that's a mistake, it's also filtering on any number of non-digits (including zero), which pretty much matches anything. Try these settings
transforms
[filter]
REGEX = ^\D+
DEST_KEY = queue
FORMAT = nullQueue
props.conf
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
CHARSET=AUTO
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-null=filter
You could also go the other way around. Do the nullQueue by default and only send to indexQueue those that _do_ match the timestamp regex.
The regex is filtering lines beginning with "^", which aren't in your sample data. Assuming that's a mistake, it's also filtering on any number of non-digits (including zero), which pretty much matches anything. Try these settings
transforms
[filter]
REGEX = ^\D+
DEST_KEY = queue
FORMAT = nullQueue
props.conf
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
CHARSET=AUTO
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-null=filter