How to generate a search to find increment user ID attempts?

[moesaidi]

I have a set of Apache access_logs where a URL is something similar to:
http://mydomain.com/user.php?userid=123

I'm trying to find any attempts where consecutive URL requests for the same URL are repeated but with increment userid values, in a short period of time.
For example, identify if this happens:

http://mydomain.com/user.php?userid=123
http://mydomain.com/user.php?userid=124
http://mydomain.com/user.php?userid=125
etc..

I haven't been able to determine the search needed to take the value of 'userid' from the last event found, increment by 1, and see if the next event found contains that userid value.

Any suggestions?

[niketn]

Since userid field is a key value pair, I would expect it to be available as extracted field in search time. Based on the same all you need to do is add delta command to compute difference between previous and current userid

 <Your Base Search>
 | delta userid as IncrementAmt

Whenever the increment in UserID is 1 are the events you want to catch.

Following is run anywhere example with mocked up data

| gentimes start="03/17/2017:01:00:00" end="03/17/2017:01:15:00" increment=1m 
| eval _time = starttime
| eval userstr= "1"
| accum userstr
| eval userid=122+userstr
| eval URI="http://mydomain.com/user.php?userid=".userid
| table _time URI
| rex field=URI "http:\/\/mydomain\.com\/user\.php\?userid=(?<userid>\d+)"
| table  _time URI userid
| delta userid as IncrementAmt

[DalJeanis]

Just as an addition to niketnilay's suggestion, it seems like a brute force attempt like this could use any increment at all.

Over any reasonable length of time, the IncrementAmt should average to zero, unless you are adding new users at a consistent and ferocious rate, in which case it will be mildly positive.

So, instead of just looking for those +1 values, you can look for trends or outliers in the increment. This run-anywhere code generates 180 pseudorandom userids.

 | gentimes start="03/17/2017:01:00:00" end="03/17/2017:04:00:01" increment=1m 
 | eval _time = starttime
 | eval userstr= "3"
 | accum userstr
 | eval userid=random() %1000 
 | eval URI="http://mydomain.com/user.php?userid=".userid
 | table _time URI
 | rex field=URI "http:\/\/mydomain\.com\/user\.php\?userid=(?<userid>\d+)"
 | table  _time URI userid
 | delta userid as IncrementAmt

This next part counts how many of each IncrementAmt have occurred. The highest count I've seen in a dozen runs is 4 of the same IncrementAmt. (180 records with random ids in the range of 0 to 999) Change the % value in the random() line to represent your actual userid range, and change the end time in the gentimes statement to whatever number of records per hour are reasonable for your installation, and see what your upper bound on the same increment might be.

 | stats count as incrementcount by IncrementAmt
 | sort 0 -incrementcount

Once you have that number, you can just set an alert for whenever the count of any particular increment is way above the norm.

Now, if you want a little more information about the overall shape of your userid usage, you could do something like this, or you could use more advanced stats functions on the underlying data.

 | eval inc2 = IncrementAmt
 | stats count as incrementcount sum(inc2) as incrementweight values(inc2) as incrementavg by IncrementAmt
 | appendpipe [| stats sum(incrementcount) as incrementcount sum(incrementweight) as incrementweight | eval incrementavg=Round(incrementweight/incrementcount,2), IncrementAmt
="Totals"] | sort 0 -incrementcount

You can also automatically scale out the increment and see what is going on across the period of the search...

| eventstats min(userid) as minuserid max(userid) as maxuserid range(userid) as rangeuserid
| eval myscale=pow(10,ceiling(.5+log(rangeuserid,10))-2)
| eval inc3=myscale*floor(IncrementAmt/myscale)
| table _time userid IncrementAmt inc3
| chart count by inc3

or over time...

| eventstats min(userid) as minuserid max(userid) as maxuserid range(userid) as rangeuserid
| eval myscale=pow(10,ceiling(.5+log(rangeuserid,10))-2)
| eval inc3=myscale*floor(IncrementAmt/myscale)
| table _time userid IncrementAmt inc3    | timechart avg(userid) avg(IncrementAmt) span=15m

[cmerriman · Accepted solution]

try something like (rex only needed if userid isn't a field already):

|rex field=_raw "userid=(?<userid>.*)"|sort 0 + _time|streamstats current=f window=1 values(userid) as previousUserid|eval diff=userid-previousUserid|search diff=1

[moesaidi]

This is the easiest solution I found; and quickest.
I liked the others as well for more advanced tactics to try, but for now this worked fine.
I'm likely going to customize it a bit to show me a chart based on what URI was attempted before, and after, with each ID and only accept 'bad attempts' as the ones using the same URI as well.