I am parsing SFTP logs of file downloads and want to count how many bytes a specific user downloaded at what time. The logs look like this (I am abbreviating the standard rfc5424 syslog prefix): session opened for local user XXX from [10.#.#.#]
received client version #
open "/some/file/name" flags READ mode 0666
close "/some/file/name" bytes read ### written #
open "/some/other/file/name" flags READ mode 0666
close "/some/other/file/name" bytes read ### written #
open "/and/another/filename" flags READ mode 0666
close "/and/another/filename" bytes read ### written #
session closed for local user XXX from [10.#.#.#] I want to somehow show how many bytes a specific user downloaded at what time. I start by inline extraction of a few extra helper fields like the username and the file sizes for example: appname=sftp-server
| rex field=_raw "session (opened|closed) for local user (?<sftp_user>[^ ]+) from"
| rex field=_raw "close \"(?<sftp_filename>.*)\" bytes read (?<sftp_bytes_read>\d+)" If I wanted to see how much data was downloaded (without caring about which user) I would just do a timechart which does the trick: appname=sftp-server
| rex field=_raw "session (opened|closed) for local user (?<sftp_user>[^ ]+) from"
| rex field=_raw "close \"(?<sftp_filename>.*)\" bytes read (?<sftp_bytes_read>\d+)"
| timechart sum(sftp_bytes_read) However the event which has the file size, does not have the user so I can filter or chart the username. If I want to filter by sftp_user, the only way I found how to do it is by making a transaction for the user session and then filtering on the sftp_user (in the example below, host, appname, and procid are extracted by the rfc5424 syslog addon): appname=sftp-server
| rex field=_raw "session (opened|closed) for local user (?<sftp_user>[^ ]+) from"
| rex field=_raw "close \"(?<sftp_filename>.*)\" bytes read (?<sftp_bytes_read>\d+)"
| transaction host appname procid sftp_user startswith="session opened for" endswith="session closed for" This does give me an events for every single SFTP session by the user, but I cannot figure out how to get the details of each file download (the individual "close" lines) out of it. What would be the way to do that, or just... "explode" it back into individual events?
... View more