How to count the difference in a rolling number in events across multiple hosts?

[Rob]

The following example events are indexed by Splunk:

Dec 1 00:47:58 serverName data-collector[1234]: #A_RECV# 1234, 5
Dec 1 00:47:58 serverName data-collector[1234]: #B_RECV# 5678, 10
Dec 1 00:47:58 serverName data-collector[1234]: #C_RECV# 2587, 100
Dec 1 00:47:58 serverName data-collector[1234]: #D_RECV# 9852, 150

What I would like to do is for a search that can show a table including the host name (A_RECV, B_RECV, etc.) the number of requests (first number) and the number of errors (second number).

Both the number of requests and the number of errors are incremental so therefore the search will need to show the difference between the number of requests from the previous event to the last event if there has been a change. An example of the events from host A might be:

Dec 1 00:47:58 serverName data-collector[1234]: #A_RECV# 1234, 5
Dec 1 00:53:58 serverName data-collector[1234]: #A_RECV# 1240, 7

If so, then the table should show that Host A had 6 requests and 2 errors.

[Rob · Accepted solution]

I believe you may want to use something similar to the following search.

host="hostName" | rex "(?P<IDLetter>[A-Z]).RECV..(?P<Requests>\\d+),.(?P<ErrorCount>\\d+)" | eval time=_time| transaction fields=IDLetter maxspan=60m | stats range(ErrorCount) range(Requests) range(time) by IDLetter

The search will first break up the string by using regex to extract the fields we want to pay attention to followed by getting the UTC time stamp and then piping that to only show the events for the last 60min. Then its just a final pipe to the stats in order to work the math to find the differences between the error count, the requests and the time.