I want to create a search that will post the amount of users that haven't finished their registration transaction. There are three events that must be shown in the log for it to be a completed transaction: "IPInterceptor", "GetPolicy", and "ActivatedNode". I'm thinking of using their IP addresses to differentiate the users and I've done this using regex and creating a field for it called ip which holds all the IP addresses of the users using my application. Here's what I have in the search so far...
blah...| rex "(?\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"
How do I iterate through the created ip field to confirm that a user has all three functions executed with their specific IP address and add up the total amount of users that did started with IPInterceptor but did not get to ActivateNode.
Raw data example:
...
[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()
...
[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()
...
[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()
...
@elijahm Please check out the following run anywhere example. You need to play with the final filter to ensure that only events matching your requirement are pulled.
| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")
As per your question the filter in the examples:
ActivateNode.doPost, GetPolicy.doPost,IPInterceptor.preHandle
GetPolicy.doPost
but do not end with ActivateNode.doPost
Following is the anywhere example
| makeresults
| eval data="[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()...[Mon Jul 29 12:23:14][INFO ][11.12.21.319][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.319][]GetPolicy.doPost()"
| makemv data delim="..."
| mvexpand data
| rename data as _raw
| rex "^\[(?<time>[^\]]+)\].*\[(?<ip>[^\]]+)\]\[\](?<type>[^\(]+)\(\)$"
| eval _time=strptime(time,"%a %b %d %H:%M:%S")
| fields - _raw time
| sort - _time
| stats count as eventCount min(_time) as _time max(_time) as latestTime list(type) as allTypes values(type) as uniqueTypes by ip
| eval duration=latestTime-_time, firstEvent=mvindex(allTypes,0), lastEvent=mvindex(allTypes,eventCount-1)
| fields - latestTime
| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")
Like this:
...| rex "(?<src_ip>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\]\[\](?<function>[^\(]+)
| streamstats count(eval(searchmatch("IPInterceptor.preHandle"))) AS sessionID BY src_ip
| stats dc(function) AS function_count values(function) AS functions BY sessionID src_ip
| stats count AS total count(eval(function_count==3) AS complete
@elijahm Please check out the following run anywhere example. You need to play with the final filter to ensure that only events matching your requirement are pulled.
| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")
As per your question the filter in the examples:
ActivateNode.doPost, GetPolicy.doPost,IPInterceptor.preHandle
GetPolicy.doPost
but do not end with ActivateNode.doPost
Following is the anywhere example
| makeresults
| eval data="[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()...[Mon Jul 29 12:23:14][INFO ][11.12.21.319][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.319][]GetPolicy.doPost()"
| makemv data delim="..."
| mvexpand data
| rename data as _raw
| rex "^\[(?<time>[^\]]+)\].*\[(?<ip>[^\]]+)\]\[\](?<type>[^\(]+)\(\)$"
| eval _time=strptime(time,"%a %b %d %H:%M:%S")
| fields - _raw time
| sort - _time
| stats count as eventCount min(_time) as _time max(_time) as latestTime list(type) as allTypes values(type) as uniqueTypes by ip
| eval duration=latestTime-_time, firstEvent=mvindex(allTypes,0), lastEvent=mvindex(allTypes,eventCount-1)
| fields - latestTime
| search (NOT (uniqueTypes="ActivateNode.doPost" AND uniqueTypes="GetPolicy.doPost" AND uniqueTypes="IPInterceptor.preHandle")) AND (lastEvent!="ActivateNode.doPost" AND firstEvent="GetPolicy.doPost")
Thank you. This got me off to a great start.
First I made some fake data with a few more events than your sample: |makeresults . . . |mvexpand data
| makeresults
| eval data="[Mon Jul 29 12:23:14][INFO ][11.12.21.318][]IPInterceptor.preHandle()...[Mon Jul 29 12:30:01][INFO ][11.12.21.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.12.21.318][]ActivateNode.doPost()...[Mon Jul 29 12:30:01][INFO ][11.12.47.318][]GetPolicy.doPost()...[Mon Jul 29 12:31:21][INFO ][11.47.21.318][]IPInterceptor.preHandle()"
| makemv delim="..." data
| mvexpand data
| rex field=data ".*\[(?<IP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\]\[\](?<activity>IPInterceptor|GetPolicy|ActivateNode).*"
| stats list(activity) as activities by IP
| where match(activities, "IPInterceptor")
| where NOT match(activities, "ActivateNode")
Then create a field for IP and a field for activity, where activity has to be one of the three items you specified: | rex field=data . . .
Then the last step is to list out which activities each IP address has passed through: | stats list(activity) as activities by IP
And finally find the ones that you are interested in: | where match(activities, "IPInterceptor") | where NOT match(activities, "ActivateNode")
Good luck, I hope this helps!
Thanks for your input. Very helpful.