topic Re: How to process varying number of Key=Value pairs in a log? in Dashboards & Visualizations

How to process varying number of Key=Value pairs in a log?

SonnyB — Sat, 07 Apr 2012 00:12:03 GMT

Scenario: Event lines in the log come with varying number of key=value pairs, where nothing is predetermined: Neither the names of the keys, nor the set of values, nor the number of such key=value pairs found in the log.

The question is: How to create generic, dynamic processing search/eval construct, that can gather and process such values from the eventdata? (i.e. only “=” and “,” delimiters are standardized).

Giving below is an example that would show the situation. Any help will be deeply appreciated!

Specification:

<< some std. access_combined fields here>> followed by: indefinite, comma-separated K=V pairs

where K=V format is: methodName=DurationInteger

Sample Log data example:

Event1: … method1=100,method2=250,method3=150

Event2: … method1=125,method2=275,method3=325,methodSome=300,method5=50

Event3: … method1=15,method2=35,methodOther=100,nextMethod4=500

Event4: … method1=125,method2Last=275

Event5: … methodSolo=400

Regex/Search/Eval expression needed to be built that can dynamically gather and sum-up all the integer numbers representing the duration values of all the above method-names without knowing the number of such key=value pairs in advance in any eventdata line (i.e. the answer should be = 3025 for the entire transaction that constitute the above five events). Any insights would be greatly appreciated. Happy Easter!

Re: How to process varying number of Key=Value pairs in a log?

Damien_Dallimor — Sat, 07 Apr 2012 06:50:44 GMT

Splunk will auto extract the key=value pairs for you at Search time.

As for adding up the method times, the following example seemed to work in 4.3. It's a bit hacky and relys on the method times being the only numeric fields extracted at Search time. But as you say, nothing is predetermined, so I can't declare a pattern in the addTotals command to detect the method fields ie: "method*"

So try this :

... | fields - linecount | addtotals fieldname=foo | stats sum(foo)

Re: How to process varying number of Key=Value pairs in a log?

gkanapathy — Sat, 07 Apr 2012 21:12:32 GMT

Yes, this is one of the main benefits of Splunk over a structured logging solution. Basically, you can just put whatever in there, and it will extract them just fine. There are some tricks or additional config if you need to worry about escaping odd characters, or have multi-line values, but there are some answers on this site that address this if you need.

Re: How to process varying number of Key=Value pairs in a log?

plynch52 — Tue, 28 Mar 2017 19:06:50 GMT

Similar issue to Sonny.
I have a a variable indeterminate number of keys (up to 1 million) some of which are in each log message and each key has an associated count. The "keys" are numeric given the large number of potential keys. There is standard fixed format prefix information that is handled correctly.

L-rec1 std-stuff[987=31,13=2, 70201=11]
Lrec2 std-stuff[91453=87,861=101,31297=76,78=1001,987=11,123=678,135=246,971=677]

Search picks up the key value pairs as "field1" to "fieldn" where there are about 900 max Kv pairs.
I want to work with the keys (group, scatter).
Any suggestions would be appreciated
:-)