Finding changes in a field

[Brad]

We are trying to watch the NIC statistics for our OS interfaces.  We are gathering data from a simple

ifconfig eth0 | grep -E 'dropped|packets' > /var/log/nic-errors.log

For my search, I have:

index="myindex" host="our-hosts*" source="/var/log/nic-errors.log"
| rex "RX\serrors\s(?<rxError>\d+)\s"
| rex "RX\spackets\s(?<rxPackets>\d+)\s"
| rex "RX\serrors\s+\d+\s+dropped\s(?<rxDrop>\d+)\s"
| chart last(rxError), last(rxPackets), last(rxDrop) by host

which displays the base data.  Now I want to watch if rxError increases and flag that.  Any ideas?
The input data will look something like:

RX packets 2165342  bytes 33209324712 (3.0 GiB)
RX errors 0  dropped 123   overruns 0   frame 0
TX packets 1988336   bytes 2848819271 (2.6 GiB)
TX errors 0   dropped 0 overruns 0  carrier 0   collisions 0

[Brad]

OK, trying:

index="myindex" host="our-hosts*" source="/var/log/nic-errors.log"
| rex "RX\serrors\s(?<rxError>\d+)\s"
| rex "RX\spackets\s(?<rxPackets>\d+)\s"
| rex "RX\serrors\s+\d+\s+dropped\s(?<rxDrop>\d+)\s"
| sort - _time
| streamstats current=f last(rxError) as priorErr last(_time) as priorTim by host
| where not (rxError=priorErr)
| chart last(rxError), last(rxPackets), last(rxDrop) by host

Will that show me when rxError changes?

[ITWhisperer]

You might also need global=false on the streamstats

| streamstats current=f global=f last(rxError) as priorErr last(_time) as priorTim by host

[Brad]

Right now I'm just running proof of concept.  I'll move the field definitions to the indexers later.  Right now I'm trying to detect

if diff pos1=last(rxError) pos2=last-1(rxError) 

I want to detect when the value or rxError changes from last-1 to last.  Working on that.

[Brad]

We are collecting every 10 minutes and have about 1000 servers with another 1000 coming early next year.  We have interest long term in monitoring all of the output for general network health.  The task at hand is being able to check if there are network issues when we also notice Ceph OSD issues.  The advice for that is to look for dropped packets on the host side.  So, that is what I'm trying to capture and detect when the dropped packet value changes.

[PickleRick]

There are some additional issues here (feel free to ignore my comments since they are a bit advanced and might be simply overkill if your case is relatively small and simple).

1. You're not using field extractions. You're extracting fields "manually" within your search. For a simple case it might work relatively well but it usually helps a lot if you have extractions defined in configuration - it lets you search for particular fields way faster than having to parse every single event and verifying if the value matches.

2. Your signal to noise ratio is relatively low - you have quite a lot of text which doesn't bring any additional value to your data - you don't have any dynamic fields so you don't have to dynamically name them and such. You could "squeeze" your events to leave only relevant values in some more structured but less verbose format. Again - if you just have a few hundred bytes each minute, that's probably not worth the work you'd need to put into it but if you have several thousands of hosts monitored this way, that could be worth savings on license costs.

3. And the most advanced topic here - you could prepare your data properly and ingest it to a metrics index. This way each event consumes a constant 160 bytes of license but most importantly - searching and doing statistical analyses over metrics indexes is way faster than on normal event indexes (but at the same time it's done a bit differently so you have to learn to use new commands like mstats or mpreview).

[ITWhisperer]

Presumably, there will some sort of time element (which you have not described). Do you collect these statistics on a regular basis? Are you events timestamped accordingly? Do you want to repeatedly search the same data to determine the last values? Have you considered running scheduled searches to collect the data in a summary index and then searching that for significant changes over time?