I have a Splunk search that contains a call to a custom search command that I created. The command (python script) takes in events (windows event log events) and finds event sequences. The command only returns the events that are part of a found sequence and removes all the other events from the results, so hundreds, thousands, or more events go into the command and typically 0 or just a few events are returned. The command also adds a few fields to each of the remaining events to denote what sequence it is part of, the step in the sequence, and the time duration into the sequence.
This search with the custom command works fine from a Splunk search page, I can see the search results on the "statistics" tab. Although, if I change the search mode from Verbose to either Fast or Smart, I no longer get my results and instead get "No results found". The results come back when I switch back to Verbose mode. When I save my search as a dashboard, the dashboard shows "No results found". If I edit the dashboard and change the visualization from "statistics table" to "events" and then back to "statistics table" and save, I see the search results on the dashboard. But, when I reload the dashboard it shows "No results found" again.
Below is the search that I am using on the search page and on the dashboard:
source="2021-06-*" index="wineventlog"
(sourcetype="WinEventLog:Security" OR sourcetype="WinEventLog:Microsoft-Windows-Sysmon/Operational")
((EventCode=11) OR (EventCode=4688 AND New_Process_Name="*schtasks.exe*")) NOT splunk
| eval TargetFilename = mvindex(TargetFilename, 0)
| evidseq F 2h
11 "(ComputerName>'name',TargetFilename>'exe')"
4688 "(New_Process_Name='*schtasks.exe*',ComputerName!'%name%',Process_Command_Line='*%exe%*')"
11 "(ComputerName='%name%')"
| table _time EventCode timeSinceStart sequenceId sequenceStep ComputerName TargetFilename New_Process_Name Process_Command_Line
| sort sequenceId sequenceStep
Also, I found that if I add the following to the bottom of my search on the dashboard, the correct results actually do appear consistently, even when I reload the dashboard, but I do not want to display my results with the transaction command.
| transaction sequenceId | sort sequenceId
So, how can I (reliably) get the results to show up on the dashboard? Any help anyone can provide would be greatly appreciated!
