topic Re: automatic nested field extraction in Splunk Search

automatic nested field extraction

mpatnode — Wed, 18 Aug 2010 03:22:59 GMT

I'm struggling with trying to extract multiple fields from a multivalue Active Directory attribute. For instance, given the following object:

dcName=w2k3r2.demo.dev
admonEventType=Update
Names:
    objectCategory=CN=Service-Connection-Point,CN=Schema,CN=Configuration,DC=demo,DC=dev
    ...
    distinguishedName=CN=bsmith,CN=Users,CN=default,CN=Zones,CN=Centrify,CN=Program Data,DC=demo,DC=dev
    objectGUID=cffb0829-0642-134c-2ef1-f03cc696e10b
          ...
    keywords=addr:253|animal:rabbit|color:blue
    showInAdvancedViewOnly=TRUE

I still want objectGUID and the other single value attributes parsed, but in this example, I also would like addr, animal and color parsed out as their own key-value pairs (also, I don't want to have to know the keynames apriori). Is there a preprocessing step where I can break the multi-value attributes into separate lines, or do I need to replace the ad-kv "(?<_KEY_1>[\w-]+)=(?<_VAL_1>[^\r\n]*)" tranform with some incredibly gnarly regex?

Re: automatic nested field extraction

Stephen_Sorkin — Wed, 18 Aug 2010 03:28:17 GMT

Do you only want this in the "keywords" field or could the pipe delimited key:value pairs occur as values of other fields as well?

Re: automatic nested field extraction

mpatnode — Wed, 18 Aug 2010 08:05:14 GMT

This was trivial, once I found the right doc

In transforms.conf

[keywords-kv]
SOURCE = keywords
DELIMS = "|", ":"

Then in my search:

sourcetype="ActiveDirectory" keywords | extract keywords-kv

So now, I'd like to do this for all ActiveDirectory objects, and handle it both in keywords or description. It would be nice if I didn't need to add the "extract" pipes.

Re: automatic nested field extraction

gkanapathy — Wed, 18 Aug 2010 08:17:47 GMT

You don't. see my other answer.

Re: automatic nested field extraction

gkanapathy — Wed, 18 Aug 2010 08:20:46 GMT

Put in props.conf:

[ActiveDirectory]
REPORT-MESSAGE = ad-kv,keywords-kv

This overrides the default extraction (which is just "ad-kv"), and listing "keywords-kv" on the same line after it ensures that it runs after the "ad-kv" has had a chance to extract the keywords field first.

Re: automatic nested field extraction

mpatnode — Wed, 18 Aug 2010 08:27:28 GMT

Much better, but now can I do this before the event is stored so the fields look like first class fields in the event? In particular, I want to detect a change to one of the keyword fields, currently diff only shows me the keyword attribute changed.

Re: automatic nested field extraction

gkanapathy — Wed, 18 Aug 2010 09:00:05 GMT

It doesn't matter whether the field is separately indexed or not. Please note that the ad-kv fields are also not extracted at index time, and are not any more "first-class" than the keyword fields. It simply appears that way because "diff" operates line-by-line against the full raw text, and the non-keyword fields happen to be on their own lines. What you really need is a field-by-field diff, which sadly Splunk does not come with.