Splunk Search

How to blacklist regex in parsing?

qcjacobo2577
Path Finder

Currently running Splunk Universal Forwarder version 9.0.3.

Looking to ignore Windows event logs (EventCode = 4103) using a "blacklist" approach as part of my overall inputs.conf configuration.  While the splunkd.log is not throwing any errors with my current attempts, it is also not ignoring logs containing the string: 

String: to Ignore: C:\WINDOWS\CCM\SystemTemp.  

Note: I am choosing to filter on the string above as other aspects can vary and this is the common string that is included in the events I want to ignore.  Below an example of such log.  Please advise.

 

My attempt at this is: 

 

blacklist1 = EventCode="4103" Message="(?:Host Application =)\s+(?:.*WINDOWS\\CCM\\SystemTemp\\+.*)"

 

 

 

User=SYSTEM
Sid=S-1-5-18
SidType=1
SourceName=Microsoft-Windows-PowerShell
Type=Information
RecordNumber=10132121
Keywords=None
TaskCategory=Executing Pipeline
OpCode=To be used when operation is just executing a method
Message=CommandInvocation(Out-Default): "Out-Default"


Context:
        Severity = Informational
        Host Name = ConsoleHost
        Host Version = 5.1.19041.2364
        Host ID = 5009593d-812d-49fc-a794-4633cf58cd5c
        Host Application = C:\WINDOWS\system32\WindowsPowerShell\v1.0\PowerShell.exe -NoLogo -Noninteractive -NoProfile -ExecutionPolicy Bypass & 'C:\WINDOWS\CCM\SystemTemp\7f1a326f-19f5-4480-9414-46ffe015e730.ps1'
        Engine Version = 5.1.19041.2364

 

 

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

https://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf#Event_Log_filtering

You have

 

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

 

Where you should have

 

blacklist1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

(yeah, I didn't check the key name myself; just checked the value before).

 

View solution in original post

qcjacobo2577
Path Finder

All --- thank you for your insight thus far.  Unfortunately, the issue still persists despite trying a number of different approaches.  The current blacklists (included in the screenshot below) are successful in that they are able to filter out events (such as those shown in the sample log --- also shown below).

Note: I do not have the Windows TA installed on the endpoint on which this blacklist is published (only upstream on my HA and Splunk Cloud hosted environment).

inputs.conf 

[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
index = windows
evt_resolve_ad_obj = 1
checkpointInterval = 5
renderXml = false
whitelist = 4103,4104

blacklist1 = EventCode="4103" "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+"
blacklist2 = EventCode="4104" "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+"

 Log

01/26/2023 09:57:53 AM
LogName=Microsoft-Windows-PowerShell/Operational
EventCode=4104
EventType=5
ComputerName=COMPUTER.acme.corp
User=SYSTEM
Sid=S-1-5-18
SidType=1
SourceName=Microsoft-Windows-PowerShell
Type=Verbose
RecordNumber=10689125
Keywords=None
TaskCategory=Execute a Remote Command
OpCode=On create calls
Message=Creating Scriptblock text (1 of 1):
{
            [void]$batch.Add($_)
            if ($batch.Count -ge $batchSize) {
                $batchAndStanza = [System.Tuple]::Create($batch.ToArray(), $stanza)
                [void]$serializingQueue.Add($batchAndStanza)
                [void]$batch.Clear()
            }
        }

ScriptBlock ID: b3e9c4de-ea6b-445a-a3af-31131f9337a7
Path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell-common.ps1

 

Where this strategy falls apart is with the log example below where the field containing this data contains other data as well (not just the file path in question).  Is there an approach here with regards to RegEx and blacklisting that will work?  If so, can you provide such?

01/26/2023 11:06:05 AM
LogName=Microsoft-Windows-PowerShell/Operational
EventCode=4103
EventType=4
ComputerName=COMPUTER.acme.corp
User=SYSTEM
Sid=S-1-5-18
SidType=1
SourceName=Microsoft-Windows-PowerShell
Type=Information
RecordNumber=10710308
Keywords=None
TaskCategory=Executing Pipeline
OpCode=To be used when operation is just executing a method
Message=CommandInvocation(Start-Sleep): "Start-Sleep"
ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"


Context:
        Severity = Informational
        Host Name = ConsoleHost
        Host Version = 5.1.19041.2364
        Host ID = 669eaf0a-e3f8-4a4a-ba5c-07aaa7cbf8bf
        Host Application = powershell.exe -command & {get-content C:\WINDOWS\TEMP\\input20898122fee20402.tmp | C:\Program` Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1  C:\Program` Files\SplunkUniversalForwarder  20898122fee20402}
        Engine Version = 5.1.19041.2364
        Runspace ID = 5291985e-c02c-4052-9465-91ab0c5a7989
        Pipeline ID = 1
        Command Name = Start-Sleep
        Command Type = Cmdlet
        Script Name = 
        Command Path = 
        Sequence Number = 47702
        User = ACME\SYSTEM
        Connected User = 
        Shell ID = Microsoft.PowerShell

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

And this blacklist works without specifying key for the regex? I'm surprised. Spec says it is key=regex format

https://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf#Event_Log_filtering

0 Karma

qcjacobo2577
Path Finder

@PickleRick Short answer is yes --- the logs for those events do not show up in an indexed search.  splunkd.log does throw an error stating what you are referring to.

 

01-26-2023 09:57:45.534 -0800 ERROR ExecProcessor [25044 ExecProcessor] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - Processing: 'blacklist1', failed to find key in string data ' "+*SplunkUniversalForwarder\\bin\\splunk-powershell*+"'

 

Any ideas as to the correct regex logic to filter the issue I am facing?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

As it is key=regex, the blacklist entry should say something like

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

 

0 Karma

qcjacobo2577
Path Finder

@PickleRick So, here is my revised inputs.conf based on your feedback:

[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
index = windows
evt_resolve_ad_obj = 1
checkpointInterval = 5
renderXml = false
whitelist = 4103,4104

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

However, this log is still being indexed:

Untitled.png

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

https://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf#Event_Log_filtering

You have

 

blacklist.1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

 

Where you should have

 

blacklist1 = EventCode="4103" Message="SplunkUniversalForwarder\\bin\\splunk-powershell"

(yeah, I didn't check the key name myself; just checked the value before).

 

qcjacobo2577
Path Finder

@PickleRick That seems to work. Thank you!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

If I remember correctly, you specify blacklist entries as as field=raw_regex, without putting the regex i  quotes. If you specify EventCode="4103", you're looking for a regex containing a quote, literal 4103 and a quote. (That's also why you might have to escape "=" if contained within a regex).

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@PickleRick The regex has to be delimited, but any character can be used, not just quotes.

---
If this reply helps you, Karma would be appreciated.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yeah, sorry. I verified it with default inputs.conf from TA-windows and you're right. I must have mistaken it with something else 😕 It's not my day, apparently..

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @qcjacobo2577,

Can you try as blacklist3 ? If you are sending Splunk_TA_windows to UF it has already blacklist1 defined.

blacklist3 = EventCode="4103" Message="(?:Host Application =)\s+(?:.*WINDOWS\\CCM\\SystemTemp\\+.*)"
If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

I'm guessing the regex isn't working for you or you wouldn't have posted here.  What I can't guess at, however, is what is happening instead.  Are all events deleted, some of them deleted, none of them deleted, or something else?

From a regex perspective, the capture groups are unnecessary as are the trailing .*.

---
If this reply helps you, Karma would be appreciated.
0 Karma

qcjacobo2577
Path Finder

@richgalloway Definitely not working as-is.  I have tried several different iterations of this using a variety of syntax.  In some cases the events continue to flow in as if I had not applied any filtering and in other cases no events with code 4103 come in at all (even though I am not targeting such a broad set of data).  I just can't seem to get it right and was hoping there was an easy answer.

Alternatively, I am open to using ingest actions --- but am less familair with using those.  Does using these only affect my license or would this approach prevent the data from being indexed at all (to whatever would have been the destination index in the first place)?  

My goal here is that these events are in the end not searchable via the destination index at all.

If ingest actions are not going to get me to my end state, I am back to using blacklists and am hoping for some ideas.

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Ingest Actions move the filtering from the UF to the indexer.  They may help or they may just move the problem to a different server.

Data filtered by an Ingest Action would not count against your license because it would not be indexed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...