Here is a link the dataset and the regex. It is working on regexr but not in transforms.conf. I have tested by using . as my regex and it then sends all logs to the nullqueue so I know the stanzas are correct, it's a problem with the regex and I have not been able to figure it out.
Here are my stanzas from props.conf and transforms.conf
props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2
Transforms.conf
[EliminateCS2]
REGEX = (?:{"ScreenshotsTakenCount".*|{"ProcessCreateFlags").*
DEST_Key = queue
FORMAT = nullQueue
Any help is appreciated.
Transforms.conf
[EliminateCS2]
REGEX = ScreenshotsTakenCount|ProcessCreateFlags
DEST_Key = queue
FORMAT = nullQueue
This is enough.
Thank you for the response. That regex is not working either.
[nullqueue_json]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TIME_PREFIX = timestamp\":\"
TRANSFORMS-CS = nullqueue_json
my test setting.
INDEXED_EXTRACTIONS=json interferes with nullqueue.
try KV_MODE=json
Thanks for the info. I am making progress but not quite there yet. I think the problem is with the line breaking. The events are being being separated properly which is causing the regex to fail.
I am guessing that I just need the proper line_breaker regex and I will be good. The end of line character in the json logs is }
I thought I could just use that as my line breaker but it's not working properly. I have tried the line breaks below.
LINE_BREAKER = }
LINE_BREAKER = ([\r\n]+)
LINE_BREAKER = (){
So now I have the line break and stanza correct as the events are finally being broken properly. The regex to send some of the events to nullqueue is still failing. I will post a sample of an event I want to go to nullqueu and see if anyone knows a regex that will catch the event and send it to nullqueue. I will also re-post my current stanzas.
props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2
TRANSFORMS-CS = EliminateCS1
KV_MODE = json
LINE_BREAKER = (){
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = false
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
TIME_PREFIX="timestamp":"
TIME_FORMAT = %s%3N TZ=UTC
pulldown_type = 1
transforms.conf
[EliminateCS1]
REGEX = event_simpleName!=EndOfProcess
DEST_Key = queue
FORMAT = nullQueue
[EliminateCS2]
REGEX = event_simpleName!=ProcessRollup2
DEST_Key = queue
FORMAT = nullQueue
Sample raw event:
{"ProcessCreateFlags":"67109888","IntegrityLevel":"16384","ParentProcessId":"33794688676116","SourceProcessId":"33794688676116","aip":"97.78.178.74","SHA1HashData":"0000000000000000000000000000000000000000","UserSid":"S-1-5-18","event_platform":"Win","TokenType":"1","ProcessEndTime":"","ParentBaseFileName":"btool.exe","ImageSubsystem":"3","id":"c3385391-dbc9-11ea-a5c6-0266311e7407","EffectiveTransmissionClass":"3","SessionId":"0","Tags":"27, 29, 40, 53, 54, 12094627905582","timestamp":"1597147019837","event_simpleName":"ProcessRollup2","RawProcessId":"6140","ConfigStateHash":"2029599784","MD5HashData":"1d5d767be226372deafbc19e716951e5","SHA256HashData":"ca3799b190ffd79c910dc0a4395b5b1fc6dacbfc2b8dbf65328d2a5ca09dec5a","ProcessSxsFlags":"64","AuthenticationId":"999","ConfigBuild":"1007.3.0011406.1","WindowFlags":"384","CommandLine":"\"E:\\Program Files\\Splunk\\bin\\SplunkD.EXE\" btool web list","ParentAuthenticationId":"999","TargetProcessId":"33794689225796","ImageFileName":"\\Device\\HarddiskVolume3\\Program Files\\Splunk\\bin\\splunkd.exe","SourceThreadId":"439906675541924","Entitlements":"15","name":"ProcessRollup2V17","ProcessStartTime":"1597147019.397","ProcessParameterFlags":"24577","aid":"8abeeb6f90da4cf3abc45b5d6fdd79cf","cid":"0396954fdb9e4990ac33e9deb40e211b"}