Problem in using append to combine correlation rules

[pt]

I am building a correlation search in Splunk ES Cloud 8 using multiple detections combined with append. Each subsearch looks for a specific lateral movement technique (e.g., suspicious scheduled task creation, Kerberos service ticket anomalies, special privileged logons, and rapid authentications).

The issue I’m running into is that when I include the “WinEvent Scheduled Task Created Within Public Path” rule (EventCode=4698) as the first search (base search), it doesn’t produce results in the final stats output, even though it returns data when run standalone. When I move it into an append subsearch (e.g., after the Kerberos tickets rule), it shows results correctly.

This is my sample search
[ search index="wineventlog" EventCode=4698 ...
| stats count by Computer, TaskName, TaskContent, SubjectUserName ]
| append
[ search index="wineventlog" EventCode=4769 ...
| bucket span=5m _time
| stats dc(ServiceName) ... by _time, IpAddress, TargetUserName ]
...
| bucket span=1h _time
| stats dc(rule_id) as rule_count ... by _time, user
Could anyone confirm if this is the right fix or suggest a better practice?”

[PickleRick]

Wait a second.

If you use 

[ search index="wineventlog" EventCode=4698 ...
| stats count by Computer, TaskName, TaskContent, SubjectUserName ]

as your "first" search, since it's not used as part of any command which requires a subsearch, it will work like this:

1) The subsearch will be spawned and run

2) It will return a set of results - since it's stats by 4 fields, each result will consist of 5 fields

3) Those results will get rendered into a set of conditions to the "main" search like

((count=X AND Computer=X AND TaskName=X AND TaskContent=X AND SubjectUserName=X) OR
(count=Y AND Computer=Y AND TaskName=Y AND TaskContent=Y AND SubjectUserName=Y) OR
... )

Since you most probably don't have a field called "count" in your original data, this search will most probably not return any results.

So you will only get results from the appended part.

That's one thing

Another thing is that you should be very very careful with subsearches. In most of their uses they have their limits and if a subsearch runs for too long or returns too many results it will be silently finalized and you will have no idea that you have been given incomplete or wrong results.

I'm not fully sure what you're trying to do here so can't offer any specific advice how to "fix" your search but if you're really trying to combine two completely different searches summary indexing might be the way to go - you run two separate searches and collect their results into an index. Finally you asynchronously run a simple search just gathering results from both of your "compound" searches and summarize them.

[ITWhisperer]

The first part doesn't need to be a subsearch

index="wineventlog" EventCode=4698 ...
| stats count by Computer, TaskName, TaskContent, SubjectUserName
| append
[ search index="wineventlog" EventCode=4769 ...
| bucket span=5m _time
| stats dc(ServiceName) ... by _time, IpAddress, TargetUserName ]
...
| bucket span=1h _time
| stats dc(rule_id) as rule_count ... by _time, user