Splunk Search

how to sort by multiple fields based on min and max and total?

Path Finder

I have a router with multiple FPCs and each FPC has multiple ICHIPs. An ICHIP can produce pktwr drops and that number of drops is logged periodically. So, within a span of time, say 720 minutes, I want to pick out the minimum number of drops (per host per FPC per ICHIP). This would be the start value for that ICHIP. Also, I want to pick out the maximum number of drops; this is the finish value. Then I want to subtract the minimum value from the maximum value to find the total. Then I want to take that total and divide by 720 minutes to get a rough drop rate. Then I want to sort the results by the highest values first.

The raw logs look like this:

Aug 11 19:01:24 a-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477315
Aug 11 19:03:20 c-priv-01.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56030
Aug 11 19:11:07 t-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477325
Aug 11 19:20:50 c-priv-03.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56130

I had put this together and used some eval and min and max statements to produce some undesirable results:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr | rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)" | timechart span=12h eval((max(err) - min(err)) / 720) by host

How do I sort by multiple fields? I am not sure if my logic is correct in the search above.

I want the results to look like this:

host          fpc    ichip    pktwr_drops     rate
c-priv-03      1       3         100           .14
t-priv-03      2       0         10            .01
Tags (1)

Splunk Employee
Splunk Employee

You don't really want to use the timechart command if you don't want one row in your table per timespan. I think that you want a search like:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr
| rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)"
| stats range(err) as pktwr_drops by host, fpc, ichip
| eval rate = pktwr_drops/720
| sort - rate

You can even be more clever by using the addinfo command to find the timebounds of the search and use that instead of the constant 720:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr
| rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)"
| stats range(err) as pktwr_drops by host, fpc, ichip
| addinfo
| eval rate = pktwr_drops/(info_max_time-info_min_time)
| fields - info*
| sort - rate

Splunk Employee
Splunk Employee

It should be as simple as:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr | rex "(?fpc\d+) (?ICHIP(\d+)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?\d+)" | timechart span=12h eval((max(err) - min(err)) / 720) by host | sort -pktwr_drops,rate

Not sure which fields you want to sort by, just giving an example.

0 Karma

Splunk Employee
Splunk Employee

Yeah, that should work if you have those fields available.

0 Karma

Path Finder

the result has to look like this

host fpc ichip pktwr_drops rate c-priv-03

1 3 100 .14 t-priv-03 2 0 10 .01

0 Karma

Path Finder

I need to sort it by highest rate, host, fpc and ichip. However rate is not defined yet. rate = (max(err) - min(err)) / 720

so can I do it like this?

  • | timechart span=12h eval rate=(max(err) - min(err))/720 by rate | sort -host,fpc,ichip ?
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!