topic Re: Different search results in search and dashboard in Splunk Search

Different search results in search and dashboard

NoSpaces — Mon, 10 Mar 2025 09:14:01 GMT

Hello everyone!
I came across a strange behavior.
I was building a dashboard and noticed that some results look unexpected.
The results are presented at the top of the screenshot.
On the last row, you can see that ProvDuration is 0.
Also, StartTime and EndTime are equal.
Moreover, other fields are also equal, and it's illogical due to the search specifics.
As you can see, StartTime and EndTime represent the min and max values of the _time field.

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP") | rex field=_raw "VM\s+(?<MachineName>.*)$" | table _time, PoolId, MachineName, _raw | transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false | search (PoolId="*") (MachineName="*") | search duration<=700 | stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId | eval DurationMin = round(DurationMin, 2) | eval DurationAvg = round(DurationAvg, 2) | eval DurationMax = round(DurationMax, 2) | eval ProvDuration = round((EndTime - StartTime), 2) | eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q") | eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q") | table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime

I decided to dig deeper and try to analyze the search more carefully.
After I moved to the search through the dashboard, I found that the search results look different.
The last row looks as it should be.
You can see these results at the bottom of the screenshot.
What could be wrong with my search, and what am I missing?

Re: Different search results in search and dashboard

kiran_panchavat — Mon, 10 Mar 2025 09:35:07 GMT

@NoSpaces

Ensure that both searches (dashboard and manual) are using the same time range.
Check the time picker settings in the dashboard. The default time range in a dashboard might be different from the one you used in the search bar.
If you have multiple panels, ensure that they are all using the same base search. Sometimes, panels might be referencing different searches, leading to inconsistencies

Re: Different search results in search and dashboard

ITWhisperer — Mon, 10 Mar 2025 11:26:30 GMT

The transaction command is returning "transactions" with only one event. Try something like this

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP") | rex field=_raw "VM\s+(?<MachineName>.*)$" | table _time, PoolId, MachineName, _raw | transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false | where eventcount > 1 | search (PoolId="*") (MachineName="*") | search duration<=700 | stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId | eval DurationMin = round(DurationMin, 2) | eval DurationAvg = round(DurationAvg, 2) | eval DurationMax = round(DurationMax, 2) | eval ProvDuration = round((EndTime - StartTime), 2) | eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q") | eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q") | table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime

Re: Different search results in search and dashboard

NoSpaces — Mon, 10 Mar 2025 11:43:18 GMT

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP") | table _time, PoolId, MachineName, _raw | rex field=_raw "VM\s+(?<MachineName>.*)$"

After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

If someone understands why transaction behavior is changing because of the table command, please supplement my answer