topic Re: Extract fields from Windows Event Log Message in Splunk Search

Extract fields from Windows Event Log Message

dw385 — Sat, 24 Jun 2017 02:55:34 GMT

I have events that do not extract the fields from the message field by default. I'm trying to setup props/transforms to auto extract the fields on search. I believe I have the props and transforms setup properly however it's not working. I think the issue is the regex I'm using and possibly the first 2 lines of the message data.

Props:

[source::WinEventLog:Directory Service]
REPORT-dsfields = ds_fields

Transforms:

[ds_fields]
SOURCE_KEY = Message
REGEX = ([^:]+):\n(.*[\S])
FORMAT = $1::$2
MV_ADD = true
KEEP_EMPTY_VALS = true

The regex appears to extract the fields properly with the exception of the first line which I don't need. I also tried this regex

(?:.*\n){2}([^:]+):\n(.*[\S])

Which also appears to work to exclude the first two lines. However the combination doesn't extract fields when searching. If I extract a single value in the same props and transforms it works so I believe the files are configured properly.

I don't care if the first line (Internal Event:) is extracted or not. The goal is to extract the field value pairs in the rest of the message.

The message field data looks like:

Internal event: A client issued a search operation with the following options. 

Client:
10.215.215.215:61759 
Starting node:
DC=domain,DC=Domain,DC=Domain,DC=local 
Filter:
 (accountNameHistory=*[DG]*)  
Search scope:
subtree 
Attribute selection:
accountNameHistory,employeeType,title,info,groupType,distinguishedName,objectGUID,sAMAccountType,departmentNumber,memberOf,objectSid,objectClass
Server controls:

Visited entries:
1556 
Returned entries:
8 
Used indexes:
Ancestors_index:3356:N; 
Pages referenced:
16745 
Pages read from disk:
31 
Pages preread from disk:
9 
Clean pages modified:
0 
Dirty pages modified:
0 
Search time (ms):
313 
Attributes Preventing Optimization:
accountNameHistory  
User:
Domain\some-user-account

Re: Extract fields from Windows Event Log Message

Richfez — Sat, 24 Jun 2017 22:19:31 GMT

I changed the text of the event code to "code" so the asterisks won't be converted to italics. It did not change the carriage returns, but I don't know if this is the way it is supposed to be. Can you confirm it still looks like the event as you see it in the original place?

Thanks!

Re: Extract fields from Windows Event Log Message

dw385 — Sun, 25 Jun 2017 01:40:28 GMT

Thank you, the example looks correct.

Re: Extract fields from Windows Event Log Message

adonio — Tue, 29 Sep 2020 14:34:43 GMT

hello there,
the configurations you need are pre-built and packaged nicely on the Splunk_TA_microsoft_ad
download here: https://splunkbase.splunk.com/app/3207/
install it on the forwarder of your domain controller and you can remove the props and transforms
read more here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/WhataSplunkAppforWindowsInfrastructuredeploymentlookslike
and here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/ConfigureActiveDirectoryauditpolicy
hope it helps

Re: Extract fields from Windows Event Log Message

dw385 — Sun, 25 Jun 2017 21:04:11 GMT

I have that deployed to my DCs and on the SHs. The data is collected but some of the logs in Directory Services is in a header over value format. It seems none of values are extracted properly (not just for this particular event I need). I don't think I see anything in the AD TA that seems to suggest there is anything for extracting the fields from the message.

Re: Extract fields from Windows Event Log Message

adonio — Sun, 25 Jun 2017 22:13:57 GMT

do you have the TA on the indexer as well?

Re: Extract fields from Windows Event Log Message

dw385 — Mon, 26 Jun 2017 10:29:42 GMT

Yes, it's on the indexers as well. Latest version (1.0) with no customization to it. This is the same for the Windows TA as well, although not the latest version, I am a release or two behind on that one.

Re: Extract fields from Windows Event Log Message

adonio — Mon, 26 Jun 2017 12:02:33 GMT

can you verify the source of the data you are trying to transform?
i see in the exchange app i have access to that the source is "Directory_Service" and not
"Directory Service"
in any case, its odd, i see all extractions working fine on the environment i have access to
hope it helps