Splunk Search

Different search results in search and dashboard

NoSpaces
Contributor

Hello everyone!
I came across a strange behavior.
I was building a dashboard and noticed that some results look unexpected.
The results are presented at the top of the screenshot.
On the last row, you can see that ProvDuration is 0.
Also, StartTime and EndTime are equal.
Moreover, other fields are also equal, and it's illogical due to the search specifics.
As you can see, StartTime and EndTime represent the min and max values of the _time field.

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime

 

I decided to dig deeper and try to analyze the search more carefully.
After I moved to the search through the dashboard, I found that the search results look different.
The last row looks as it should be.
You can see these results at the bottom of the screenshot.
What could be wrong with my search, and what am I missing?

Labels (2)
0 Karma
1 Solution

NoSpaces
Contributor

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

View solution in original post

0 Karma

NoSpaces
Contributor

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The transaction command is returning "transactions" with only one event. Try something like this

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| where eventcount > 1
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime
0 Karma

kiran_panchavat
Champion

@NoSpaces 

  • Ensure that both searches (dashboard and manual) are using the same time range.
  • Check the time picker settings in the dashboard. The default time range in a dashboard might be different from the one you used in the search bar.
  • If you have multiple panels, ensure that they are all using the same base search. Sometimes, panels might be referencing different searches, leading to inconsistencies
Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...

Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend ...

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward ...

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

Ready to make your IT operations smarter and more efficient? Discover how to automate Splunk alerts with Red ...