topic Re: Dynamically extract field names from multiline event in Splunk Search

Dynamically extract field names from multiline event

frink — Fri, 29 Apr 2011 17:00:55 GMT

I've got some log data that has a multi-line event this format:

Thanks the the help of others on this forum, I can now pull each of the key-value pairs from the SPARAM rows, but I have to use one field extract per possible key:

... | rex field=_raw "(?m-s)^SPARAM\|\d*\|PartNumber\|(?<SearchPartNumber>.*)"

Is it possible to write one extract that would give me all the keys as different fields? I've got about 20 possible keys, and I want to make this extract future-proof as well?

Can I write something that will give me "PartNumber", "OtherParameter" and "OtherParameter2" as field names?

Thanks.

Re: Dynamically extract field names from multiline event

Ledion_Bitincka — Fri, 29 Apr 2011 17:41:23 GMT

A couple of things:

(1) I would not recommend using rex to do field extractions (unless you're just testing stuff), but rather configure automatic field extraction in props/transforms.conf (maybe you're just testing ... )

(2) you can extract field name and field value from the event (note that you cannot modify the field name as you're doing PartNumber -> SearchPartNumber though)

props.conf
[my_sourcetype]
...
REPORT-fields = my_fields

transforms.conf
[my_fields]
REGEX = (?m-s)^SPARAM\|\d*\|([^|]+)\|(.*)
FORMAT = $1::$2

Re: Dynamically extract field names from multiline event

frink — Fri, 29 Apr 2011 18:37:47 GMT

Thanks for the quick response. Is there a way to do it using rex?

I'm not the administrator of this system so it will be more difficult for me to get the properties file changed (probably coming with a working proof of concept will help).

Thanks.

Re: Dynamically extract field names from multiline event

Ledion_Bitincka — Fri, 29 Apr 2011 18:52:43 GMT

No, there is no way to do this with rex. However, you can configure field extractions from the Manage, if you're using 4.2 you should be able to configure the above via:
Manager » Fields » Field transformations and
Manager » Fields » Field extractions

Re: Dynamically extract field names from multiline event

frink — Fri, 29 Apr 2011 19:23:26 GMT

Thanks, I'll give that a shot.

Re: Dynamically extract field names from multiline event

bojanz — Sat, 30 Apr 2011 10:39:18 GMT

Actually, there is benefit in using rex. If you configure automatic field extraction in props/transforms it will be applied by Splunk to every search result for that particular source type - and regular expressions can be very expensive.

If you use rex, you can filter search so they are applied to a much smaller result set.

Re: Dynamically extract field names from multiline event

Ledion_Bitincka — Sat, 30 Apr 2011 23:15:00 GMT

That is not completely true. Splunk applies the field extraction only to events that are pulled from the index - NOT all events in a sourcetype. So, if you're able to filter events before rex you should also be able to filter them as part of the first search. However, there are corner cases where the first search is not able to filter results before field extractions

Re: Dynamically extract field names from multiline event

khourihan_splun — Fri, 18 Apr 2014 23:43:50 GMT

another trick if you are experiencing performance issues, (I am find issues using the expanded-snare-syslog app) is to run the search in fast mode and add the fields you want.

i.e. search | fields fieldA fieldB etc..