Getting Data In

Nullqueue not working

byeb1264
Explorer

Here is a link the dataset and the regex.  It is working on regexr but not in transforms.conf.  I have tested by using . as my regex and it then sends all logs to the nullqueue so I know the stanzas are correct, it's a problem with the regex and I have not been able to figure it out.

https://regexr.com/59qu2

Here are my stanzas from props.conf and transforms.conf

props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2

 

Transforms.conf
[EliminateCS2]
REGEX = (?:{"ScreenshotsTakenCount".*|{"ProcessCreateFlags").*
DEST_Key = queue
FORMAT = nullQueue

Any help is appreciated.  

Labels (2)
0 Karma

to4kawa
Ultra Champion

Transforms.conf
[EliminateCS2]
REGEX = ScreenshotsTakenCount|ProcessCreateFlags
DEST_Key = queue
FORMAT = nullQueue

This is enough.

0 Karma

byeb1264
Explorer

Thank you for the response.  That regex is not working either.  

0 Karma

to4kawa
Ultra Champion

[nullqueue_json]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TIME_PREFIX = timestamp\":\"
TRANSFORMS-CS = nullqueue_json

my test setting.

INDEXED_EXTRACTIONS=json interferes with nullqueue.
try KV_MODE=json

0 Karma

byeb1264
Explorer

Thanks for the info.  I am making progress but not quite there yet.  I think the problem is with the line breaking.  The events are being being separated properly which is causing the regex to fail.

I am guessing that I just need the proper line_breaker regex and I will be good.  The end of line character in the json logs is } 

I thought I could just use that as my line breaker but it's not working properly.  I have tried the line breaks below.

LINE_BREAKER = }
LINE_BREAKER = ([\r\n]+)

0 Karma

to4kawa
Ultra Champion

LINE_BREAKER = (){


byeb1264
Explorer

So now I have the line break and stanza correct as the events are finally being broken properly.  The regex to send some of the events to nullqueue is still failing.  I will post a sample of an event I want to go to nullqueu and see if anyone knows a regex that will catch the event and send it to nullqueue.  I will also re-post my current stanzas.

props.conf

[cs_replicator]
TRANSFORMS-CS = EliminateCS2
TRANSFORMS-CS = EliminateCS1
KV_MODE = json
LINE_BREAKER = (){
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = false
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
TIME_PREFIX="timestamp":"
TIME_FORMAT = %s%3N TZ=UTC
pulldown_type = 1

transforms.conf

[EliminateCS1]
REGEX = event_simpleName!=EndOfProcess
DEST_Key = queue
FORMAT = nullQueue

[EliminateCS2]
REGEX = event_simpleName!=ProcessRollup2
DEST_Key = queue
FORMAT = nullQueue

 

Sample raw event:

{"ProcessCreateFlags":"67109888","IntegrityLevel":"16384","ParentProcessId":"33794688676116","SourceProcessId":"33794688676116","aip":"97.78.178.74","SHA1HashData":"0000000000000000000000000000000000000000","UserSid":"S-1-5-18","event_platform":"Win","TokenType":"1","ProcessEndTime":"","ParentBaseFileName":"btool.exe","ImageSubsystem":"3","id":"c3385391-dbc9-11ea-a5c6-0266311e7407","EffectiveTransmissionClass":"3","SessionId":"0","Tags":"27, 29, 40, 53, 54, 12094627905582","timestamp":"1597147019837","event_simpleName":"ProcessRollup2","RawProcessId":"6140","ConfigStateHash":"2029599784","MD5HashData":"1d5d767be226372deafbc19e716951e5","SHA256HashData":"ca3799b190ffd79c910dc0a4395b5b1fc6dacbfc2b8dbf65328d2a5ca09dec5a","ProcessSxsFlags":"64","AuthenticationId":"999","ConfigBuild":"1007.3.0011406.1","WindowFlags":"384","CommandLine":"\"E:\\Program Files\\Splunk\\bin\\SplunkD.EXE\" btool web list","ParentAuthenticationId":"999","TargetProcessId":"33794689225796","ImageFileName":"\\Device\\HarddiskVolume3\\Program Files\\Splunk\\bin\\splunkd.exe","SourceThreadId":"439906675541924","Entitlements":"15","name":"ProcessRollup2V17","ProcessStartTime":"1597147019.397","ProcessParameterFlags":"24577","aid":"8abeeb6f90da4cf3abc45b5d6fdd79cf","cid":"0396954fdb9e4990ac33e9deb40e211b"}
0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...

Updated Data Management and AWS GDI Inventory in Splunk Observability

We’re making some changes to Data Management and Infrastructure Inventory for AWS. The Data Management page, ...