topic Re: How to extract field:value pairs from within the value of other field? in Splunk Search

How to extract field:value pairs from within the value of other field?

att35 — Fri, 12 Aug 2016 15:18:51 GMT

Hi,

We are capturing a custom log from Windows event viewer using Splunk forwarder. Most of the fields are extracted properly in field:value format, except one where the data itself contains sub-fields with their own values.

I tried creating reg-ex field extractions but it only identified in 1 or 2 events and keeps failing for other.
e.g. Below is a sanitized version of the event we see. All fields from LogName to Keywords are parsed correctly, except when it reached Message. This field itself contains other items e.g. Client, Computer, or Action.

What is the best way to extract these fields as well?

08/12/2016 10:23:03 AM
LogName=xxxxxxxxx
SourceName=xxxxxxxxx
EventCode=3
EventType=4
Type=Information
ComputerName=Name.x.y.z.local
User=aaaa
Sid=S-1-5-7
SidType=5
TaskCategory=%1
OpCode=None
RecordNumber=1234567
Keywords=Classic
Message=AD object was successfully modified.
        Client Computer : w.x.y.z
        Object DN : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local
        Object Class : user
        Object GUID : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local     Attribute Name : logonCount
        Action : Update
        Old Value : 1234
        New Value : 1235
        Request ID : {84E5C0C7-84E5C0C7-84E5C0C7-84E5C0C7}

Re: How to extract field:value pairs from within the value of other field?

skoelpin — Tue, 29 Sep 2020 10:35:52 GMT

You can do this

index=foo | rex field=ORIGINAL_FILED_NAME "(?<NEW_FILED_NAME>\w+)"

This will do a regular expression and extract data from your first field (ORIGINAL_FIELD_NAME) and create a second field (NEW_FILED_NAME) with the extracted value

I'm not sure what second field you want extracted, give us more details and I'll write the regex for you

Re: How to extract field:value pairs from within the value of other field?

gcusello — Fri, 12 Aug 2016 15:44:53 GMT

If you want to extract only the string after message= but not the other lines, you have to insert \n at the end of your regex, see below:
Message=(?.*)\n
if you want to try this regex use https://regex101.com/
Bye.
Giuseppe

Re: How to extract field:value pairs from within the value of other field?

att35 — Fri, 12 Aug 2016 15:52:20 GMT

Hi skoelpin,

Thank you for the response.

From the "Message" field in above example, we would like to extract all the sub-field, i.e. Client Computer, Object DN, Object Class, Object GUID, Action, Old Value, New Value and Request ID. It will become much easier to create dashboards and alerts if all these are their own field:value pair instead of being part of original "Message" field.

Re: How to extract field:value pairs from within the value of other field?

gcusello — Fri, 12 Aug 2016 15:57:23 GMT

it's the same thing :
Client Computer : (? .)\n
Object DN : (? .)\n
Object Class : (? .)\n
Object GUID : (? .)\n
Action : (? .)\n
Old Value : (? .)\n
New Value : (? .)\n
Request ID : (? .)
You have to extract every field inserting \n at the end
Bye.
Giuseppe

Re: How to extract field:value pairs from within the value of other field?

skoelpin — Fri, 12 Aug 2016 16:00:40 GMT

I would recommend extracting these sub-fields at search time in your props.conf

So you would go to your props.conf and go to your stanza and add this

[YOUR_SOURCETYPE]
EXTRACT-GUID = ((?<=CN\=|OU\=|DC\=)\w+)

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles

Re: How to extract field:value pairs from within the value of other field?

sundareshr — Fri, 12 Aug 2016 17:04:40 GMT

If you looking to extract this in the search screen, you could use the extract command.

... | extract kvdelim=" : " pairdelim="\n"

http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract

Re: How to extract field:value pairs from within the value of other field?

woodcock — Mon, 15 Aug 2016 15:17:45 GMT

You can install the Windows_TA app and it should do field extractions for you.

Re: How to extract field:value pairs from within the value of other field?

att35 — Tue, 29 Sep 2020 10:45:55 GMT

Hi skoelpin,

I tried this but didn't work. Just to make sure, this needs to be added under the system/local/props.conf correct? because we also have Splunk_TA_windows addon installed which has it's own props.conf.

I also came across another answer but again, none of the changes take effect.
https://answers.splunk.com/answers/49310/field-extraction-in-message-field-of-windows-event-log.html

I am not sure If I am editing/adding extractions at the wrong location? or if it's because the source/sourcetype contains white spaces, although I tried adding a wildcard * but in vain.
source=[WinEventLog:XXX Y Z]

Re: How to extract field:value pairs from within the value of other field?

att35 — Tue, 29 Sep 2020 10:45:57 GMT

Hi woodcock,

We have Splunk_TA_windows installed on both the Search head as well as the forwarder sending these events, but the Message field still the same. Entire Message is clubbed into one big string.

We also tried moving the monitor stanza on the universal forwarder from the system/local/inputs.conf to inputs.conf for windows TA, but it still behaves the same way.

~ Abhi

Re: How to extract field:value pairs from within the value of other field?

att35 — Fri, 26 Aug 2016 16:08:28 GMT

Based on the fields currently being extracted for windows event's, it looks like the props from Windows TA is contributing as well. I am not sure if this takes precedence over /local/props.conf or if both work together.

Is there a way to figure out which extractions/transforms and from which location are being used for a specific sourcetype.

Re: How to extract field:value pairs from within the value of other field?

tred23 — Mon, 27 Feb 2017 12:46:08 GMT

I'm having the same issue with the windows TA installed as well.

Re: How to extract field:value pairs from within the value of other field?

woodcock — Tue, 28 Feb 2017 21:35:08 GMT

Let's make sure that we are talking about the same things; see here:

https://answers.splunk.com/answers/504719/splunk-for-windows-app-on-standalone-instance.html