Getting Data In

Nullqueue not working

byeb1264
Explorer

Here is a link the dataset and the regex.  It is working on regexr but not in transforms.conf.  I have tested by using . as my regex and it then sends all logs to the nullqueue so I know the stanzas are correct, it's a problem with the regex and I have not been able to figure it out.

https://regexr.com/59qu2

Here are my stanzas from props.conf and transforms.conf

props.conf
[cs_replicator]
TRANSFORMS-CS = EliminateCS2

 

Transforms.conf
[EliminateCS2]
REGEX = (?:{"ScreenshotsTakenCount".*|{"ProcessCreateFlags").*
DEST_Key = queue
FORMAT = nullQueue

Any help is appreciated.  

Labels (2)
0 Karma

to4kawa
Ultra Champion

Transforms.conf
[EliminateCS2]
REGEX = ScreenshotsTakenCount|ProcessCreateFlags
DEST_Key = queue
FORMAT = nullQueue

This is enough.

0 Karma

byeb1264
Explorer

Thank you for the response.  That regex is not working either.  

0 Karma

to4kawa
Ultra Champion

[nullqueue_json]
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
TIME_PREFIX = timestamp\":\"
TRANSFORMS-CS = nullqueue_json

my test setting.

INDEXED_EXTRACTIONS=json interferes with nullqueue.
try KV_MODE=json

0 Karma

byeb1264
Explorer

Thanks for the info.  I am making progress but not quite there yet.  I think the problem is with the line breaking.  The events are being being separated properly which is causing the regex to fail.

I am guessing that I just need the proper line_breaker regex and I will be good.  The end of line character in the json logs is } 

I thought I could just use that as my line breaker but it's not working properly.  I have tried the line breaks below.

LINE_BREAKER = }
LINE_BREAKER = ([\r\n]+)

0 Karma

to4kawa
Ultra Champion

LINE_BREAKER = (){


byeb1264
Explorer

So now I have the line break and stanza correct as the events are finally being broken properly.  The regex to send some of the events to nullqueue is still failing.  I will post a sample of an event I want to go to nullqueu and see if anyone knows a regex that will catch the event and send it to nullqueue.  I will also re-post my current stanzas.

props.conf

[cs_replicator]
TRANSFORMS-CS = EliminateCS2
TRANSFORMS-CS = EliminateCS1
KV_MODE = json
LINE_BREAKER = (){
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = false
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
TIME_PREFIX="timestamp":"
TIME_FORMAT = %s%3N TZ=UTC
pulldown_type = 1

transforms.conf

[EliminateCS1]
REGEX = event_simpleName!=EndOfProcess
DEST_Key = queue
FORMAT = nullQueue

[EliminateCS2]
REGEX = event_simpleName!=ProcessRollup2
DEST_Key = queue
FORMAT = nullQueue

 

Sample raw event:

{"ProcessCreateFlags":"67109888","IntegrityLevel":"16384","ParentProcessId":"33794688676116","SourceProcessId":"33794688676116","aip":"97.78.178.74","SHA1HashData":"0000000000000000000000000000000000000000","UserSid":"S-1-5-18","event_platform":"Win","TokenType":"1","ProcessEndTime":"","ParentBaseFileName":"btool.exe","ImageSubsystem":"3","id":"c3385391-dbc9-11ea-a5c6-0266311e7407","EffectiveTransmissionClass":"3","SessionId":"0","Tags":"27, 29, 40, 53, 54, 12094627905582","timestamp":"1597147019837","event_simpleName":"ProcessRollup2","RawProcessId":"6140","ConfigStateHash":"2029599784","MD5HashData":"1d5d767be226372deafbc19e716951e5","SHA256HashData":"ca3799b190ffd79c910dc0a4395b5b1fc6dacbfc2b8dbf65328d2a5ca09dec5a","ProcessSxsFlags":"64","AuthenticationId":"999","ConfigBuild":"1007.3.0011406.1","WindowFlags":"384","CommandLine":"\"E:\\Program Files\\Splunk\\bin\\SplunkD.EXE\" btool web list","ParentAuthenticationId":"999","TargetProcessId":"33794689225796","ImageFileName":"\\Device\\HarddiskVolume3\\Program Files\\Splunk\\bin\\splunkd.exe","SourceThreadId":"439906675541924","Entitlements":"15","name":"ProcessRollup2V17","ProcessStartTime":"1597147019.397","ProcessParameterFlags":"24577","aid":"8abeeb6f90da4cf3abc45b5d6fdd79cf","cid":"0396954fdb9e4990ac33e9deb40e211b"}
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Observability Simplified: Combining User Experience, Application Performance & ...

Tech Talk Observability Simplified: Combining User Experience, Application Performance & Network ...

Event Series May & June: From Network Visibility to Service Intelligence

Unifying the Network: Moving from Alert Noise to Service Intelligence with Splunk ITSI In today’s hybrid ...