Splunk Search

how to sort by multiple fields based on min and max and total?

Path Finder

I have a router with multiple FPCs and each FPC has multiple ICHIPs. An ICHIP can produce pktwr drops and that number of drops is logged periodically. So, within a span of time, say 720 minutes, I want to pick out the minimum number of drops (per host per FPC per ICHIP). This would be the start value for that ICHIP. Also, I want to pick out the maximum number of drops; this is the finish value. Then I want to subtract the minimum value from the maximum value to find the total. Then I want to take that total and divide by 720 minutes to get a rough drop rate. Then I want to sort the results by the highest values first.

The raw logs look like this:

Aug 11 19:01:24 a-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477315
Aug 11 19:03:20 c-priv-01.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56030
Aug 11 19:11:07 t-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477325
Aug 11 19:20:50 c-priv-03.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56130

I had put this together and used some eval and min and max statements to produce some undesirable results:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr | rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)" | timechart span=12h eval((max(err) - min(err)) / 720) by host

How do I sort by multiple fields? I am not sure if my logic is correct in the search above.

I want the results to look like this:

host          fpc    ichip    pktwr_drops     rate
c-priv-03      1       3         100           .14
t-priv-03      2       0         10            .01
Tags (1)

Splunk Employee
Splunk Employee

You don't really want to use the timechart command if you don't want one row in your table per timespan. I think that you want a search like:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr
| rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)"
| stats range(err) as pktwr_drops by host, fpc, ichip
| eval rate = pktwr_drops/720
| sort - rate

You can even be more clever by using the addinfo command to find the timebounds of the search and use that instead of the constant 720:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr
| rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)"
| stats range(err) as pktwr_drops by host, fpc, ichip
| addinfo
| eval rate = pktwr_drops/(info_max_time-info_min_time)
| fields - info*
| sort - rate

Splunk Employee
Splunk Employee

It should be as simple as:

source="/var/logs/current/juniper" Packet drop in Ichip pktwr | rex "(?fpc\d+) (?ICHIP(\d+)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?\d+)" | timechart span=12h eval((max(err) - min(err)) / 720) by host | sort -pktwr_drops,rate

Not sure which fields you want to sort by, just giving an example.

0 Karma

Splunk Employee
Splunk Employee

Yeah, that should work if you have those fields available.

0 Karma

Path Finder

the result has to look like this

host fpc ichip pktwr_drops rate c-priv-03

1 3 100 .14 t-priv-03 2 0 10 .01

0 Karma

Path Finder

I need to sort it by highest rate, host, fpc and ichip. However rate is not defined yet. rate = (max(err) - min(err)) / 720

so can I do it like this?

  • | timechart span=12h eval rate=(max(err) - min(err))/720 by rate | sort -host,fpc,ichip ?
0 Karma