Dashboard creation based on iterating through IP addresses

[elijahm]

I want to create a search that will post the amount of users that haven't finished their registration transaction. There are three events that must be shown in the log for it to be a completed transaction: "IPInterceptor", "GetPolicy", and "ActivatedNode". I'm thinking of using their IP addresses to differentiate the users and I've done this using regex and creating a field for it called ip which holds all the IP addresses of the users using my application. Here's what I have in the search so far...

blah...| rex "(?\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"

How do I iterate through the created ip field to confirm that a user has all three functions executed with their specific IP address and add up the total amount of users that did started with IPInterceptor but did not get to ActivateNode.

Raw data example:

...
[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()
...
[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()
...
[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()
...

[woodcock]

Like this:

...| rex "(?<src_ip>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\]\[\](?<function>[^\(]+)
| streamstats count(eval(searchmatch("IPInterceptor.preHandle"))) AS sessionID BY src_ip
| stats dc(function) AS function_count values(function) AS functions BY sessionID src_ip
| stats count AS total count(eval(function_count==3) AS complete

[niketn · Accepted solution]

@elijahm Please check out the following run anywhere example. You need to play with the final filter to ensure that only events matching your requirement are pulled.

| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")

As per your question the filter in the examples:

1. looks for Events which do not have all three types i.e. ActivateNode.doPost, GetPolicy.doPost,IPInterceptor.preHandle
2. looks for events which start with GetPolicy.doPost but do not end with ActivateNode.doPost

Following is the anywhere example

| makeresults 
| eval data="[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()...[Mon Jul 29 12:23:14][INFO ][11.12.21.319][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.319][]GetPolicy.doPost()" 
| makemv data delim="..." 
| mvexpand data 
| rename data as _raw 
| rex "^\[(?<time>[^\]]+)\].*\[(?<ip>[^\]]+)\]\[\](?<type>[^\(]+)\(\)$" 
| eval _time=strptime(time,"%a %b %d %H:%M:%S") 
| fields - _raw time 
| sort - _time 
| stats count as eventCount min(_time) as _time max(_time) as latestTime list(type) as allTypes values(type) as uniqueTypes by ip 
| eval duration=latestTime-_time, firstEvent=mvindex(allTypes,0), lastEvent=mvindex(allTypes,eventCount-1) 
| fields - latestTime 
| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")

[elijahm]

Thank you. This got me off to a great start.

[grittonc]

First I made some fake data with a few more events than your sample: |makeresults . . . |mvexpand data

| makeresults 
| eval data="[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()...[Mon Jul 29 12:30:01][INFO ][11.12.47.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.47.21.318][]IPInterceptor.preHandle()"
| makemv delim="..." data
| mvexpand data
| rex field=data ".*\[(?<IP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\]\[\](?<activity>IPInterceptor|GetPolicy|ActivateNode).*"
| stats list(activity) as activities by IP
| where match(activities, "IPInterceptor") 
| where NOT match(activities, "ActivateNode")

Then create a field for IP and a field for activity, where activity has to be one of the three items you specified: | rex field=data . . .

Then the last step is to list out which activities each IP address has passed through: | stats list(activity) as activities by IP

And finally find the ones that you are interested in: | where match(activities, "IPInterceptor") | where NOT match(activities, "ActivateNode")

Good luck, I hope this helps!

[elijahm]

Thanks for your input. Very helpful.