Dynamically extract field names from multiline event

[frink]

I've got some log data that has a multi-line event this format:

2011-04-28 11:40:00|ACTION|1304005199906869|stuff|stuff|stuff

SPARAM|1304005199906869|PartNumber|1613034

SPARAM|1304005199906869|OtherParameter|8528

SPARAM|1304005199906869|OtherParameter2|true

Thanks the the help of others on this forum, I can now pull each of the key-value pairs from the SPARAM rows, but I have to use one field extract per possible key:

... | rex field=_raw "(?m-s)^SPARAM\|\d*\|PartNumber\|(?<SearchPartNumber>.*)"

Is it possible to write one extract that would give me all the keys as different fields? I've got about 20 possible keys, and I want to make this extract future-proof as well?

Can I write something that will give me "PartNumber", "OtherParameter" and "OtherParameter2" as field names?

Thanks.

[Ledion_Bitincka]

A couple of things:

(1) I would not recommend using rex to do field extractions (unless you're just testing stuff), but rather configure automatic field extraction in props/transforms.conf (maybe you're just testing ... )

(2) you can extract field name and field value from the event (note that you cannot modify the field name as you're doing PartNumber -> SearchPartNumber though)

props.conf
[my_sourcetype]
...
REPORT-fields = my_fields

transforms.conf
[my_fields]
REGEX = (?m-s)^SPARAM\|\d*\|([^|]+)\|(.*)
FORMAT = $1::$2

[khourihan_splun]

another trick if you are experiencing performance issues, (I am find issues using the expanded-snare-syslog app) is to run the search in fast mode and add the fields you want.

i.e. search | fields fieldA fieldB etc..

[Ledion_Bitincka]

That is not completely true. Splunk applies the field extraction only to events that are pulled from the index - NOT all events in a sourcetype. So, if you're able to filter events before rex you should also be able to filter them as part of the first search. However, there are corner cases where the first search is not able to filter results before field extractions

[bojanz]

Actually, there is benefit in using rex. If you configure automatic field extraction in props/transforms it will be applied by Splunk to every search result for that particular source type - and regular expressions can be very expensive.

If you use rex, you can filter search so they are applied to a much smaller result set.

[frink]

Thanks, I'll give that a shot.

[Ledion_Bitincka]

No, there is no way to do this with rex. However, you can configure field extractions from the Manage, if you're using 4.2 you should be able to configure the above via:
Manager » Fields » Field transformations and
Manager » Fields » Field extractions

[frink]

Thanks for the quick response. Is there a way to do it using rex?

I'm not the administrator of this system so it will be more difficult for me to get the properties file changed (probably coming with a working proof of concept will help).

Thanks.