Unable to use nullqueue properly for a Storage Account input

att35 — Thu, 27 Feb 2025 19:40:42 GMT

Hi everyone,

We are pulling Firewall data from a Storage Account containing several categories. There is one specific category, AZFWDnsQuery which need to be dropped.

I tested the regex in the search as well as on regex101. It is successfully matching only those specific events with this category.

But once deployed, Splunk starts dropping all events from this input, including for other categories that do not match the regex.

Sample events

{ "time": "2025-02-27T18:46:08.307710+00:00", "resourceId": "/SUBSCRIPTIONS/xxxxxx/xxx/Path", "properties": {"SourceIp":"x.x.x.x","SourcePort":25208,"QueryId":51787,"QueryType":"A","QueryClass":"IN","QueryName":"google.com","Protocol":"udp","RequestSize":48,"DnssecOkBit":false,"EDNS0BufferSize":512,"ResponseCode":"NOERROR","ResponseFlags":"qr,rd,ra","ResponseSize":94,"RequestDurationSecs":0.007257565,"ErrorNumber":0,"ErrorMessage":""}, "category": "AZFWDnsQuery"} { "time": "2025-02-27T18:46:08.307329+00:00", "resourceId": "/SUBSCRIPTIONS/xxxxxx/xxx/Path", "properties": {"SourceIp":"x.x.x.x","SourcePort":62730,"QueryId":16828,"QueryType":"A","QueryClass":"IN","QueryName":"google.com","Protocol":"udp","RequestSize":35,"DnssecOkBit":false,"EDNS0BufferSize":512,"ResponseCode":"NOERROR","ResponseFlags":"qr,rd,ra","ResponseSize":68,"RequestDurationSecs":0.012227477,"ErrorNumber":0,"ErrorMessage":""}, "category": "AZFWDnsQuery"} { "time": "2025-02-27T18:46:08.307262+00:00", "resourceId": "/SUBSCRIPTIONS/xxxxxx/xxx/Path", "properties": {"SourceIp":"x.x.x.x","SourcePort":45452,"QueryId":25241,"QueryType":"A","QueryClass":"IN","QueryName":"google.com","Protocol":"udp","RequestSize":35,"DnssecOkBit":false,"EDNS0BufferSize":512,"ResponseCode":"NOERROR","ResponseFlags":"qr,rd,ra","ResponseSize":68,"RequestDurationSecs":0.008439891,"ErrorNumber":0,"ErrorMessage":""}, "category": "AZFWDnsQuery"} { "time": "2025-02-27T18:46:08.307129+00:00", "resourceId": "/SUBSCRIPTIONS/xxxxxx/xxx/Path", "properties": {"SourceIp":"x.x.x.x","SourcePort":14846,"QueryId":3916,"QueryType":"A","QueryClass":"IN","QueryName":"google.com","Protocol":"udp","RequestSize":35,"DnssecOkBit":false,"EDNS0BufferSize":512,"ResponseCode":"NOERROR","ResponseFlags":"qr,rd,ra","ResponseSize":68,"RequestDurationSecs":0.009026804,"ErrorNumber":0,"ErrorMessage":""}, "category": "AZFWDnsQuery"}

Regex

\"category\":\s\"AZFWDnsQuery\"

Here is how props.conf and transforms.conf are configured.

[sourcetype] TRANSFORMS-null=DropFirewallEvents [DropFirewallEvents] REGEX=_raw=\"category\":\s\"AZFWDnsQuery\" DEST_KEY=queue FORMAT=nullQueue

What could be doing wrong here for Splunk to drop every event from this input?

Thanks

Re: Unable to use nullqueue properly for a Storage Account input

richgalloway — Thu, 27 Feb 2025 20:26:35 GMT

The REGEX attribute must contain a capturing group, even if it's not used. Also, no need to reference _raw.

[DropFirewallEvents] REGEX= ("category":\s"AZFWDnsQuery") DEST_KEY=queue FORMAT=nullQueue

Re: Unable to use nullqueue properly for a Storage Account input

att35 — Thu, 27 Feb 2025 21:38:11 GMT

Thanks @richgalloway

This worked.

topic Unable to use nullqueue properly for a Storage Account input in Getting Data In

Unable to use nullqueue properly for a Storage Account input

Re: Unable to use nullqueue properly for a Storage Account input

Re: Unable to use nullqueue properly for a Storage Account input