Charting diff of events with running total

[rtadams89]

I have some firewall session state logs which get sent to Splunk every minute. The session state events contain a unique ID per session and a running total of the bytes transferred for the duration of the session. 

Example events:

Nov  7 09:00:18 em1 tcp 23.246.39.150:443 <- 192.168.10.104:60241 6966 bytes id: 5fa81441
Nov  7 09:01:18 em1 tcp 23.246.39.150:443 <- 192.168.10.104:60241 7174 bytes id: 5fa81441
Nov  7 09:02:18 em1 tcp 23.246.39.150:443 <- 192.168.10.104:60241 8900 bytes id: 5fa81441
Nov  7 09:03:18 em1 tcp 23.246.39.150:443 <- 192.168.10.104:60241 11501 bytes id: 5fa81441

Ultimately, I would like to do a timechart showing the bytes transferred by minute. To make that work, I need to add a new field to each event that is the difference between the bytes value in the event and the bytes value in the previous event. Note, I cannot use the "delta" command directly, as the 4 events shown are mixed in with hundreds of other events and are not directly next to each other. I need like a delta combined with transaction command. I'm currently considering using transaction to group the events, and then mvexpand to break them back apart again, but my search is getting real clunky. Any recommendations on how to do the above?

[richgalloway]

You can use the delta or streamstats command if you first make your search select only relevant events.  Doing that will also improve the performance of the search.  Something like this:

index=foo "tcp" "bytes" "id:"
| rex "\s(?<bytes>\d+) bytes"
| rex "id: (?<id>\S+)"
| streamstats window=2 range(bytes) as diff by id

[rtadams89]

I'm not seeing how that would work. What I essentially need to get to is a way to transform the events presented (and note, that is just one state; there would be thousands of these intermixed) into a table that contains: timestamp, source IP, bytes transferred, id

"bytes transferred" would be equal to the total bytes reported in the original event at that timestamp, minus the total bytes transferred in the preceding event that had the same id value.