Splunk Search

How to extract field:value pairs from within the value of other field?

Contributor

Hi,

We are capturing a custom log from Windows event viewer using Splunk forwarder. Most of the fields are extracted properly in field:value format, except one where the data itself contains sub-fields with their own values.

I tried creating reg-ex field extractions but it only identified in 1 or 2 events and keeps failing for other.
e.g. Below is a sanitized version of the event we see. All fields from LogName to Keywords are parsed correctly, except when it reached Message. This field itself contains other items e.g. Client, Computer, or Action.

What is the best way to extract these fields as well?

08/12/2016 10:23:03 AM
LogName=xxxxxxxxx
SourceName=xxxxxxxxx
EventCode=3
EventType=4
Type=Information
ComputerName=Name.x.y.z.local
User=aaaa
Sid=S-1-5-7
SidType=5
TaskCategory=%1
OpCode=None
RecordNumber=1234567
Keywords=Classic
Message=AD object was successfully modified.
        Client Computer : w.x.y.z
        Object DN : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local
        Object Class : user
        Object GUID : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local     Attribute Name : logonCount
        Action : Update
        Old Value : 1234
        New Value : 1235
        Request ID : {84E5C0C7-84E5C0C7-84E5C0C7-84E5C0C7}
0 Karma

Esteemed Legend

You can install the Windows_TA app and it should do field extractions for you.

0 Karma

Contributor

Hi woodcock,

We have Splunk_TA_windows installed on both the Search head as well as the forwarder sending these events, but the Message field still the same. Entire Message is clubbed into one big string.

We also tried moving the monitor stanza on the universal forwarder from the system/local/inputs.conf to inputs.conf for windows TA, but it still behaves the same way.

~ Abhi

0 Karma

Path Finder

I'm having the same issue with the windows TA installed as well.

0 Karma

Esteemed Legend

Let's make sure that we are talking about the same things; see here:

https://answers.splunk.com/answers/504719/splunk-for-windows-app-on-standalone-instance.html

0 Karma

Legend

If you looking to extract this in the search screen, you could use the extract command.

... | extract kvdelim=" : " pairdelim="\n"

http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract

SplunkTrust
SplunkTrust

If you want to extract only the string after message= but not the other lines, you have to insert \n at the end of your regex, see below:
Message=(?.*)\n
if you want to try this regex use https://regex101.com/
Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

it's the same thing :
Client Computer : (? .)\n
Object DN : (? .
)\n
Object Class : (? .)\n
Object GUID : (? .
)\n
Action : (? .)\n
Old Value : (? .
)\n
New Value : (? .)\n
Request ID : (? .
)
You have to extract every field inserting \n at the end
Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

You can do this

index=foo | rex field=ORIGINAL_FILED_NAME "(?<NEW_FILED_NAME>\w+)"

This will do a regular expression and extract data from your first field (ORIGINAL_FIELD_NAME) and create a second field (NEW_FILED_NAME) with the extracted value

I'm not sure what second field you want extracted, give us more details and I'll write the regex for you

0 Karma

Contributor

Hi skoelpin,

Thank you for the response.

From the "Message" field in above example, we would like to extract all the sub-field, i.e. Client Computer, Object DN, Object Class, Object GUID, Action, Old Value, New Value and Request ID. It will become much easier to create dashboards and alerts if all these are their own field:value pair instead of being part of original "Message" field.

0 Karma

SplunkTrust
SplunkTrust

I would recommend extracting these sub-fields at search time in your props.conf

So you would go to your props.conf and go to your stanza and add this

[YOUR_SOURCETYPE]
EXTRACT-GUID = ((?<=CN\=|OU\=|DC\=)\w+)

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Createandmaintainsearch-timefieldextract...

0 Karma

Contributor

Hi skoelpin,

I tried this but didn't work. Just to make sure, this needs to be added under the system/local/props.conf correct? because we also have Splunk_TA_windows addon installed which has it's own props.conf.

I also came across another answer but again, none of the changes take effect.
https://answers.splunk.com/answers/49310/field-extraction-in-message-field-of-windows-event-log.html

I am not sure If I am editing/adding extractions at the wrong location? or if it's because the source/sourcetype contains white spaces, although I tried adding a wildcard * but in vain.
source=[WinEventLog:XXX Y Z]

0 Karma

Contributor

Based on the fields currently being extracted for windows event's, it looks like the props from Windows TA is contributing as well. I am not sure if this takes precedence over /local/props.conf or if both work together.

Is there a way to figure out which extractions/transforms and from which location are being used for a specific sourcetype.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!