topic Re: Splunk Blacklist RegEx Parsing in Splunk Search

How to blacklist regex in parsing?

qcjacobo2577 — Fri, 27 Jan 2023 16:53:57 GMT

Currently running Splunk Universal Forwarder version 9.0.3.

Looking to ignore Windows event logs (EventCode = 4103) using a "blacklist" approach as part of my overall inputs.conf configuration. While the splunkd.log is not throwing any errors with my current attempts, it is also not ignoring logs containing the string:

String: to Ignore: C:\WINDOWS\CCM\SystemTemp.

Note: I am choosing to filter on the string above as other aspects can vary and this is the common string that is included in the events I want to ignore. Below an example of such log. Please advise.

My attempt at this is:

blacklist1 = EventCode="4103" Message="(?:Host Application =)\s+(?:.*WINDOWS\\CCM\\SystemTemp\\+.*)"

User=SYSTEM Sid=S-1-5-18 SidType=1 SourceName=Microsoft-Windows-PowerShell Type=Information RecordNumber=10132121 Keywords=None TaskCategory=Executing Pipeline OpCode=To be used when operation is just executing a method Message=CommandInvocation(Out-Default): "Out-Default" Context: Severity = Informational Host Name = ConsoleHost Host Version = 5.1.19041.2364 Host ID = 5009593d-812d-49fc-a794-4633cf58cd5c Host Application = C:\WINDOWS\system32\WindowsPowerShell\v1.0\PowerShell.exe -NoLogo -Noninteractive -NoProfile -ExecutionPolicy Bypass & 'C:\WINDOWS\CCM\SystemTemp\7f1a326f-19f5-4480-9414-46ffe015e730.ps1' Engine Version = 5.1.19041.2364

Re: Splunk Blacklist RegEx Parsing

richgalloway — Thu, 26 Jan 2023 01:20:40 GMT

I'm guessing the regex isn't working for you or you wouldn't have posted here. What I can't guess at, however, is what is happening instead. Are all events deleted, some of them deleted, none of them deleted, or something else?

From a regex perspective, the capture groups are unnecessary as are the trailing .*.

Re: Splunk Blacklist RegEx Parsing

qcjacobo2577 — Thu, 26 Jan 2023 02:22:09 GMT

@richgalloway Definitely not working as-is. I have tried several different iterations of this using a variety of syntax. In some cases the events continue to flow in as if I had not applied any filtering and in other cases no events with code 4103 come in at all (even though I am not targeting such a broad set of data). I just can't seem to get it right and was hoping there was an easy answer.

Alternatively, I am open to using ingest actions --- but am less familair with using those. Does using these only affect my license or would this approach prevent the data from being indexed at all (to whatever would have been the destination index in the first place)?

My goal here is that these events are in the end not searchable via the destination index at all.

If ingest actions are not going to get me to my end state, I am back to using blacklists and am hoping for some ideas.

Re: Splunk Blacklist RegEx Parsing

scelikok — Thu, 26 Jan 2023 05:26:03 GMT

Hi @qcjacobo2577,

Can you try as blacklist3 ? If you are sending Splunk_TA_windows to UF it has already blacklist1 defined.

blacklist3 = EventCode="4103" Message="(?:Host Application =)\s+(?:.*WINDOWS\\CCM\\SystemTemp\\+.*)"

Re: Splunk Blacklist RegEx Parsing

richgalloway — Thu, 26 Jan 2023 13:28:31 GMT

Ingest Actions move the filtering from the UF to the indexer. They may help or they may just move the problem to a different server.

Data filtered by an Ingest Action would not count against your license because it would not be indexed.

Re: Splunk Blacklist RegEx Parsing

PickleRick — Thu, 26 Jan 2023 13:43:20 GMT

If I remember correctly, you specify blacklist entries as as field=raw_regex, without putting the regex i quotes. If you specify EventCode="4103", you're looking for a regex containing a quote, literal 4103 and a quote. (That's also why you might have to escape "=" if contained within a regex).

Re: Splunk Blacklist RegEx Parsing

richgalloway — Thu, 26 Jan 2023 13:55:27 GMT

@PickleRick The regex has to be delimited, but any character can be used, not just quotes.

Re: Splunk Blacklist RegEx Parsing

PickleRick — Thu, 26 Jan 2023 14:47:09 GMT

Yeah, sorry. I verified it with default inputs.conf from TA-windows and you're right. I must have mistaken it with something else 😕 It's not my day, apparently..

Re: Splunk Blacklist RegEx Parsing

qcjacobo2577 — Thu, 26 Jan 2023 19:08:21 GMT

All --- thank you for your insight thus far. Unfortunately, the issue still persists despite trying a number of different approaches. The current blacklists (included in the screenshot below) are successful in that they are able to filter out events (such as those shown in the sample log --- also shown below).

Note: I do not have the Windows TA installed on the endpoint on which this blacklist is published (only upstream on my HA and Splunk Cloud hosted environment).

inputs.conf

[WinEventLog://Microsoft-Windows-PowerShell/Operational] disabled = 0 index = windows evt_resolve_ad_obj = 1 checkpointInterval = 5 renderXml = false whitelist = 4103,4104 blacklist1 = EventCode="4103" "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+" blacklist2 = EventCode="4104" "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+"

Log

01/26/2023 09:57:53 AM LogName=Microsoft-Windows-PowerShell/Operational EventCode=4104 EventType=5 ComputerName=COMPUTER.acme.corp User=SYSTEM Sid=S-1-5-18 SidType=1 SourceName=Microsoft-Windows-PowerShell Type=Verbose RecordNumber=10689125 Keywords=None TaskCategory=Execute a Remote Command OpCode=On create calls Message=Creating Scriptblock text (1 of 1): { [void]$batch.Add($_) if ($batch.Count -ge $batchSize) { $batchAndStanza = [System.Tuple]::Create($batch.ToArray(), $stanza) [void]$serializingQueue.Add($batchAndStanza) [void]$batch.Clear() } } ScriptBlock ID: b3e9c4de-ea6b-445a-a3af-31131f9337a7 Path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell-common.ps1

Where this strategy falls apart is with the log example below where the field containing this data contains other data as well (not just the file path in question). Is there an approach here with regards to RegEx and blacklisting that will work? If so, can you provide such?

01/26/2023 11:06:05 AM LogName=Microsoft-Windows-PowerShell/Operational EventCode=4103 EventType=4 ComputerName=COMPUTER.acme.corp User=SYSTEM Sid=S-1-5-18 SidType=1 SourceName=Microsoft-Windows-PowerShell Type=Information RecordNumber=10710308 Keywords=None TaskCategory=Executing Pipeline OpCode=To be used when operation is just executing a method Message=CommandInvocation(Start-Sleep): "Start-Sleep" ParameterBinding(Start-Sleep): name="Milliseconds"; value="200" Context: Severity = Informational Host Name = ConsoleHost Host Version = 5.1.19041.2364 Host ID = 669eaf0a-e3f8-4a4a-ba5c-07aaa7cbf8bf Host Application = powershell.exe -command & {get-content C:\WINDOWS\TEMP\\input20898122fee20402.tmp | C:\Program` Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1 C:\Program` Files\SplunkUniversalForwarder 20898122fee20402} Engine Version = 5.1.19041.2364 Runspace ID = 5291985e-c02c-4052-9465-91ab0c5a7989 Pipeline ID = 1 Command Name = Start-Sleep Command Type = Cmdlet Script Name = Command Path = Sequence Number = 47702 User = ACME\SYSTEM Connected User = Shell ID = Microsoft.PowerShell

Re: Splunk Blacklist RegEx Parsing

PickleRick — Thu, 26 Jan 2023 19:37:25 GMT

And this blacklist works without specifying key for the regex? I'm surprised. Spec says it is key=regex format

https://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf#Event_Log_filtering

Re: Splunk Blacklist RegEx Parsing

qcjacobo2577 — Thu, 26 Jan 2023 20:11:59 GMT

@PickleRick Short answer is yes --- the logs for those events do not show up in an indexed search. splunkd.log does throw an error stating what you are referring to.

01-26-2023 09:57:45.534 -0800 ERROR ExecProcessor [25044 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - Processing: 'blacklist1', failed to find key in string data ' "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+"'

Any ideas as to the correct regex logic to filter the issue I am facing?

Re: Splunk Blacklist RegEx Parsing

PickleRick — Thu, 26 Jan 2023 20:44:32 GMT

As it is key=regex, the blacklist entry should say something like

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

Re: Splunk Blacklist RegEx Parsing

qcjacobo2577 — Thu, 26 Jan 2023 21:40:39 GMT

@PickleRick So, here is my revised inputs.conf based on your feedback:

[WinEventLog://Microsoft-Windows-PowerShell/Operational] disabled = 0 index = windows evt_resolve_ad_obj = 1 checkpointInterval = 5 renderXml = false whitelist = 4103,4104 blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

However, this log is still being indexed:

Re: Splunk Blacklist RegEx Parsing

PickleRick — Fri, 27 Jan 2023 08:52:13 GMT

https://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf#Event_Log_filtering

You have

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

Where you should have

blacklist1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

(yeah, I didn't check the key name myself; just checked the value before).

Re: Splunk Blacklist RegEx Parsing

qcjacobo2577 — Fri, 27 Jan 2023 15:55:48 GMT

@PickleRick That seems to work. Thank you!