Comparing Two Searches (WIth Different Search Patterns). Find Difference From First Search

[tdotcspot]

Hi there,

I'm having a bit of trouble trying to accomplish this and I'm hoping someone can help. I'm using the AWS add on in Splunk (which grabs CloudTrail logs) and I'm trying to create a search that shows how many EC2 instances are currently active for the day in our environment. My thought process was to 1) find the number of running instances, get the ids 2) get the number of terminated instances, find those ids and 3) compare the two searches and do a one way diff. Iterating thru each id on the 1st search and if they are not found in the ids of the 2nd search, the instance is still running.

First query:

index="cloudtrail" eventName=RunInstances  earliest=-0h@d  | spath output="runningInstances" "responseElements.instancesSet.items{}.instanceId" |table runningInstances

Second query:

 index="*" eventName=TerminateInstances earliest=-0h@d  | spath output="terminatedInstances" "responseElements.instancesSet.items{}.instanceId" |table terminatedInstances

I thought I could use a "|set diff" but don't have a lot of experience with it. I also thought I could also attempt some type of "|foreach" loop within another "|foreach" loop.. but I think I'm making it too complex.

Any help would be greatly appreciated!
Thanks,
T

[renjith_nair · Accepted solution]

@tdotcspot ,

Try,

• Get the instances from both eventNames
• Exclude the instances which are part of both event names(RunInstances and TerminateInstances)

• Filter running instances.

  index=* (eventName=RunInstances OR eventName=TerminateInstances) earliest=-0h@d
  |spath output="instances" "responseElements.instancesSet.items{}.instanceId" 
  |stats count,values(eventName) as eventNames by instances
  |where mvcount(eventNames ) <2 AND eventNames="RunInstances"
  

OR

• Get all running instances and filter them if they are found in the terminated instances.

  index="cloudtrail" eventName=RunInstances earliest=-0h@d
  
  |spath output="instances" "responseElements.instancesSet.items{}.instanceId"|fields instances
  |search NOT [ search index="*" eventName=TerminateInstances earliest=-0h@d
  
  | spath output="instances" "responseElements.instancesSet.items{}.instanceId" |fields instances]