topic Nullqueue not working in Getting Data In

Nullqueue not working

byeb1264 — Fri, 07 Aug 2020 18:08:38 GMT

Here is a link the dataset and the regex. It is working on regexr but not in transforms.conf. I have tested by using . as my regex and it then sends all logs to the nullqueue so I know the stanzas are correct, it's a problem with the regex and I have not been able to figure it out.

https://regexr.com/59qu2

Here are my stanzas from props.conf and transforms.conf

props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2

Transforms.conf
[EliminateCS2]
REGEX = (?:{"ScreenshotsTakenCount".*|{"ProcessCreateFlags").*
DEST_Key = queue
FORMAT = nullQueue

Any help is appreciated.

Re: Nullqueue not working

to4kawa — Fri, 07 Aug 2020 18:27:04 GMT

Transforms.conf
[EliminateCS2]
REGEX = ScreenshotsTakenCount|ProcessCreateFlags
DEST_Key = queue
FORMAT = nullQueue

This is enough.

Re: Nullqueue not working

byeb1264 — Fri, 07 Aug 2020 18:45:36 GMT

Thank you for the response. That regex is not working either.

Re: Nullqueue not working

to4kawa — Fri, 07 Aug 2020 19:08:03 GMT

[nullqueue_json]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TIME_PREFIX = timestamp\":\"
TRANSFORMS-CS = nullqueue_json

my test setting.

INDEXED_EXTRACTIONS=json interferes with nullqueue.
try KV_MODE=json

Re: Nullqueue not working

byeb1264 — Fri, 07 Aug 2020 20:50:07 GMT

Thanks for the info. I am making progress but not quite there yet. I think the problem is with the line breaking. The events are being being separated properly which is causing the regex to fail.

I am guessing that I just need the proper line_breaker regex and I will be good. The end of line character in the json logs is }

I thought I could just use that as my line breaker but it's not working properly. I have tried the line breaks below.

LINE_BREAKER = }
LINE_BREAKER = ([\r\n]+)

Re: Nullqueue not working

to4kawa — Fri, 07 Aug 2020 23:44:25 GMT

LINE_BREAKER = (){

Re: Nullqueue not working

byeb1264 — Tue, 11 Aug 2020 12:09:30 GMT

So now I have the line break and stanza correct as the events are finally being broken properly. The regex to send some of the events to nullqueue is still failing. I will post a sample of an event I want to go to nullqueu and see if anyone knows a regex that will catch the event and send it to nullqueue. I will also re-post my current stanzas.

props.conf

[cs_replicator]
TRANSFORMS-CS = EliminateCS2
TRANSFORMS-CS = EliminateCS1
KV_MODE = json
LINE_BREAKER = (){
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = false
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
TIME_PREFIX="timestamp":"
TIME_FORMAT = %s%3N TZ=UTC
pulldown_type = 1

transforms.conf

[EliminateCS1]
REGEX = event_simpleName!=EndOfProcess
DEST_Key = queue
FORMAT = nullQueue

[EliminateCS2]
REGEX = event_simpleName!=ProcessRollup2
DEST_Key = queue
FORMAT = nullQueue

Sample raw event:

{"ProcessCreateFlags":"67109888","IntegrityLevel":"16384","ParentProcessId":"33794688676116","SourceProcessId":"33794688676116","aip":"97.78.178.74","SHA1HashData":"0000000000000000000000000000000000000000","UserSid":"S-1-5-18","event_platform":"Win","TokenType":"1","ProcessEndTime":"","ParentBaseFileName":"btool.exe","ImageSubsystem":"3","id":"c3385391-dbc9-11ea-a5c6-0266311e7407","EffectiveTransmissionClass":"3","SessionId":"0","Tags":"27, 29, 40, 53, 54, 12094627905582","timestamp":"1597147019837","event_simpleName":"ProcessRollup2","RawProcessId":"6140","ConfigStateHash":"2029599784","MD5HashData":"1d5d767be226372deafbc19e716951e5","SHA256HashData":"ca3799b190ffd79c910dc0a4395b5b1fc6dacbfc2b8dbf65328d2a5ca09dec5a","ProcessSxsFlags":"64","AuthenticationId":"999","ConfigBuild":"1007.3.0011406.1","WindowFlags":"384","CommandLine":"\"E:\\Program Files\\Splunk\\bin\\SplunkD.EXE\" btool web list","ParentAuthenticationId":"999","TargetProcessId":"33794689225796","ImageFileName":"\\Device\\HarddiskVolume3\\Program Files\\Splunk\\bin\\splunkd.exe","SourceThreadId":"439906675541924","Entitlements":"15","name":"ProcessRollup2V17","ProcessStartTime":"1597147019.397","ProcessParameterFlags":"24577","aid":"8abeeb6f90da4cf3abc45b5d6fdd79cf","cid":"0396954fdb9e4990ac33e9deb40e211b"}