Getting Data In

How to report on the top 10 fields when logs have a variable number of key value pairs that are numeric?

Explorer

I have log records with a variable number of KV (key value) pairs. Both the field and the values are numeric. The following search parses all of the log records correctly and builds a very long row of results given that there are thousands of potential key values to sum.
I have a transforms.conf & props.conf setup

[root@splunk local]# cat transforms.conf 
[get_rule_stat]
REGEX = ([0-9]+)=([0-9]+)
FORMAT = $1::$2
MV_ADD =true
CLEAN_KEYS = false

[get_rule_id]
REGEX =(?<_KEY_1>[0-9]+)=(?<_VAL_1>[0-9]+)
MV_ADD = true
CLEAN_KEYS = false
--------
[root@splunk local]# cat props.conf 
[ScoutShield]
KV_MODE = multi
REPORT-Stats = get_rule_stat
REPORT-R-ID = get_rule_id
---




source="qa_debug.log" host="splunk.localdomain" sourcetype="ScoutShield" 
| rex field=_raw "(?ms)(?=[^N]*(?:NetDefender Rule Hits Digest|N.*NetDefender Rule Hits Digest))^(?P[^\\[]+)[^\\]\\n]*\\]\\[(?P[^\\]]+)\\]\\[(?P\\d+)\\]\\[(?P\\d+)[^\\]\\n]*\\]\\[(?P[^\\]]+)" offset_field=_extracted_fields_bounds 
| rex field=stats max_match=100 "(?\d*=\d*)" 
|stats sum(*)  
| addtotals   fieldname=TotalHits 
| rename sum(*) as * 

What I would like to do is report on the top fields (as in top 10 with largest count) or to do a timechart with only those fields where the fields belong to the top 10.

It seems as though I can only use * for the field names, and there isn't a way to sub set the names.
Any suggestions would be appreciated.

0 Karma
1 Solution

Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues

View solution in original post

0 Karma

Revered Legend

Assuming your field extraction is working fine and you're getting fields with numeric names, give this a try

your base search | table _time 1* 2* 3* 4* 5* 6* 7* 8* 9* 0*
| untable _time fieldnames fieldvalues

Once you get one row for each field using above search, your run searches like this

Top 10 fields based on count

...above search.. | top fieldnames 

Top 10 fields based on fieldvalues column

...above search.. | sort 10 -fieldvalues

Top 10 based on sum of fieldvalues

...above search.. | stats sum(fieldvalues) as fieldvalues by fieldnames | sort 10 -fieldvalues

View solution in original post

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!