Splunk Search

Different search results in search and dashboard

NoSpaces
Communicator

Hello everyone!
I came across a strange behavior.
I was building a dashboard and noticed that some results look unexpected.
The results are presented at the top of the screenshot.
On the last row, you can see that ProvDuration is 0.
Also, StartTime and EndTime are equal.
Moreover, other fields are also equal, and it's illogical due to the search specifics.
As you can see, StartTime and EndTime represent the min and max values of the _time field.

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime

 

I decided to dig deeper and try to analyze the search more carefully.
After I moved to the search through the dashboard, I found that the search results look different.
The last row looks as it should be.
You can see these results at the bottom of the screenshot.
What could be wrong with my search, and what am I missing?

Labels (2)
0 Karma
1 Solution

NoSpaces
Communicator

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

View solution in original post

0 Karma

NoSpaces
Communicator

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The transaction command is returning "transactions" with only one event. Try something like this

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| where eventcount > 1
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime
0 Karma

kiran_panchavat
Influencer

@NoSpaces 

  • Ensure that both searches (dashboard and manual) are using the same time range.
  • Check the time picker settings in the dashboard. The default time range in a dashboard might be different from the one you used in the search bar.
  • If you have multiple panels, ensure that they are all using the same base search. Sometimes, panels might be referencing different searches, leading to inconsistencies
I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.
0 Karma
Get Updates on the Splunk Community!

New This Month - Splunk Observability updates and improvements for faster ...

What’s New? This month, we’re delivering several enhancements across Splunk Observability Cloud for faster and ...

What's New in Splunk Cloud Platform 9.3.2411?

Hey Splunky People! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2411. This release ...

Buttercup Games: Further Dashboarding Techniques (Part 6)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...