Splunk Search

Extract fields from Windows Event Log Message

dw385
Explorer

I have events that do not extract the fields from the message field by default. I'm trying to setup props/transforms to auto extract the fields on search. I believe I have the props and transforms setup properly however it's not working. I think the issue is the regex I'm using and possibly the first 2 lines of the message data.

Props:

[source::WinEventLog:Directory Service]
REPORT-dsfields = ds_fields

Transforms:

[ds_fields]
SOURCE_KEY = Message
REGEX = ([^:]+):\n(.*[\S])
FORMAT = $1::$2
MV_ADD = true
KEEP_EMPTY_VALS = true

The regex appears to extract the fields properly with the exception of the first line which I don't need. I also tried this regex

(?:.*\n){2}([^:]+):\n(.*[\S])

Which also appears to work to exclude the first two lines. However the combination doesn't extract fields when searching. If I extract a single value in the same props and transforms it works so I believe the files are configured properly.

I don't care if the first line (Internal Event:) is extracted or not. The goal is to extract the field value pairs in the rest of the message.

The message field data looks like:

Internal event: A client issued a search operation with the following options. 

Client:
10.215.215.215:61759 
Starting node:
DC=domain,DC=Domain,DC=Domain,DC=local 
Filter:
 (accountNameHistory=*[DG]*)  
Search scope:
subtree 
Attribute selection:
accountNameHistory,employeeType,title,info,groupType,distinguishedName,objectGUID,sAMAccountType,departmentNumber,memberOf,objectSid,objectClass
Server controls:

Visited entries:
1556 
Returned entries:
8 
Used indexes:
Ancestors_index:3356:N; 
Pages referenced:
16745 
Pages read from disk:
31 
Pages preread from disk:
9 
Clean pages modified:
0 
Dirty pages modified:
0 
Search time (ms):
313 
Attributes Preventing Optimization:
accountNameHistory  
User:
Domain\some-user-account
Tags (1)
0 Karma

adonio
Ultra Champion

hello there,
the configurations you need are pre-built and packaged nicely on the Splunk_TA_microsoft_ad
download here: https://splunkbase.splunk.com/app/3207/
install it on the forwarder of your domain controller and you can remove the props and transforms
read more here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/WhataSplunkAppforWindowsInfrastructuredeplo...
and here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/ConfigureActiveDirectoryauditpolicy
hope it helps

0 Karma

dw385
Explorer

I have that deployed to my DCs and on the SHs. The data is collected but some of the logs in Directory Services is in a header over value format. It seems none of values are extracted properly (not just for this particular event I need). I don't think I see anything in the AD TA that seems to suggest there is anything for extracting the fields from the message.

0 Karma

adonio
Ultra Champion

do you have the TA on the indexer as well?

0 Karma

dw385
Explorer

Yes, it's on the indexers as well. Latest version (1.0) with no customization to it. This is the same for the Windows TA as well, although not the latest version, I am a release or two behind on that one.

0 Karma

adonio
Ultra Champion

can you verify the source of the data you are trying to transform?
i see in the exchange app i have access to that the source is "Directory_Service" and not
"Directory Service"
in any case, its odd, i see all extractions working fine on the environment i have access to
hope it helps

0 Karma

Richfez
SplunkTrust
SplunkTrust

I changed the text of the event code to "code" so the asterisks won't be converted to italics. It did not change the carriage returns, but I don't know if this is the way it is supposed to be. Can you confirm it still looks like the event as you see it in the original place?

Thanks!

0 Karma

dw385
Explorer

Thank you, the example looks correct.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...