Getting Data In

Nullqueue not working

byeb1264
Explorer

Here is a link the dataset and the regex.  It is working on regexr but not in transforms.conf.  I have tested by using . as my regex and it then sends all logs to the nullqueue so I know the stanzas are correct, it's a problem with the regex and I have not been able to figure it out.

https://regexr.com/59qu2

Here are my stanzas from props.conf and transforms.conf

props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2

 

Transforms.conf
[EliminateCS2]
REGEX = (?:{"ScreenshotsTakenCount".*|{"ProcessCreateFlags").*
DEST_Key = queue
FORMAT = nullQueue

Any help is appreciated.  

Labels (2)
0 Karma

to4kawa
Ultra Champion

Transforms.conf
[EliminateCS2]
REGEX = ScreenshotsTakenCount|ProcessCreateFlags
DEST_Key = queue
FORMAT = nullQueue

This is enough.

0 Karma

byeb1264
Explorer

Thank you for the response.  That regex is not working either.  

0 Karma

to4kawa
Ultra Champion

[nullqueue_json]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TIME_PREFIX = timestamp\":\"
TRANSFORMS-CS = nullqueue_json

my test setting.

INDEXED_EXTRACTIONS=json interferes with nullqueue.
try KV_MODE=json

0 Karma

byeb1264
Explorer

Thanks for the info.  I am making progress but not quite there yet.  I think the problem is with the line breaking.  The events are being being separated properly which is causing the regex to fail.

I am guessing that I just need the proper line_breaker regex and I will be good.  The end of line character in the json logs is } 

I thought I could just use that as my line breaker but it's not working properly.  I have tried the line breaks below.

LINE_BREAKER = }
LINE_BREAKER = ([\r\n]+)

0 Karma

to4kawa
Ultra Champion

LINE_BREAKER = (){


byeb1264
Explorer

So now I have the line break and stanza correct as the events are finally being broken properly.  The regex to send some of the events to nullqueue is still failing.  I will post a sample of an event I want to go to nullqueu and see if anyone knows a regex that will catch the event and send it to nullqueue.  I will also re-post my current stanzas.

props.conf

[cs_replicator]
TRANSFORMS-CS = EliminateCS2
TRANSFORMS-CS = EliminateCS1
KV_MODE = json
LINE_BREAKER = (){
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = false
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
TIME_PREFIX="timestamp":"
TIME_FORMAT = %s%3N TZ=UTC
pulldown_type = 1

transforms.conf

[EliminateCS1]
REGEX = event_simpleName!=EndOfProcess
DEST_Key = queue
FORMAT = nullQueue

[EliminateCS2]
REGEX = event_simpleName!=ProcessRollup2
DEST_Key = queue
FORMAT = nullQueue

 

Sample raw event:

{"ProcessCreateFlags":"67109888","IntegrityLevel":"16384","ParentProcessId":"33794688676116","SourceProcessId":"33794688676116","aip":"97.78.178.74","SHA1HashData":"0000000000000000000000000000000000000000","UserSid":"S-1-5-18","event_platform":"Win","TokenType":"1","ProcessEndTime":"","ParentBaseFileName":"btool.exe","ImageSubsystem":"3","id":"c3385391-dbc9-11ea-a5c6-0266311e7407","EffectiveTransmissionClass":"3","SessionId":"0","Tags":"27, 29, 40, 53, 54, 12094627905582","timestamp":"1597147019837","event_simpleName":"ProcessRollup2","RawProcessId":"6140","ConfigStateHash":"2029599784","MD5HashData":"1d5d767be226372deafbc19e716951e5","SHA256HashData":"ca3799b190ffd79c910dc0a4395b5b1fc6dacbfc2b8dbf65328d2a5ca09dec5a","ProcessSxsFlags":"64","AuthenticationId":"999","ConfigBuild":"1007.3.0011406.1","WindowFlags":"384","CommandLine":"\"E:\\Program Files\\Splunk\\bin\\SplunkD.EXE\" btool web list","ParentAuthenticationId":"999","TargetProcessId":"33794689225796","ImageFileName":"\\Device\\HarddiskVolume3\\Program Files\\Splunk\\bin\\splunkd.exe","SourceThreadId":"439906675541924","Entitlements":"15","name":"ProcessRollup2V17","ProcessStartTime":"1597147019.397","ProcessParameterFlags":"24577","aid":"8abeeb6f90da4cf3abc45b5d6fdd79cf","cid":"0396954fdb9e4990ac33e9deb40e211b"}
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...