Splunk Search

Different search results in search and dashboard

NoSpaces
Contributor

Hello everyone!
I came across a strange behavior.
I was building a dashboard and noticed that some results look unexpected.
The results are presented at the top of the screenshot.
On the last row, you can see that ProvDuration is 0.
Also, StartTime and EndTime are equal.
Moreover, other fields are also equal, and it's illogical due to the search specifics.
As you can see, StartTime and EndTime represent the min and max values of the _time field.

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime

 

I decided to dig deeper and try to analyze the search more carefully.
After I moved to the search through the dashboard, I found that the search results look different.
The last row looks as it should be.
You can see these results at the bottom of the screenshot.
What could be wrong with my search, and what am I missing?

Labels (2)
0 Karma
1 Solution

NoSpaces
Contributor

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

View solution in original post

0 Karma

NoSpaces
Contributor

UPD:
I found the solution
On the dashboard, I used a base search
It seems that the "table" command cuts something important for the transaction command

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"

 


After a dozen manual attempts, I found that results can vary without changing the time span
Then I narrowed down the search to only one machine name to analyze the transaction behavior
Eventually, I observed that the transaction remains open even if start and end events exist
Then I replaced table with fields, and the transaction started to work as expected
All in all, the working variant is:

 

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| table _time, PoolId, MachineName, _raw
| rex field=_raw "VM\s+(?<MachineName>.*)$"
<..>

 


If someone understands why transaction behavior is changing because of the table command, please supplement my answer

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The transaction command is returning "transactions" with only one event. Try something like this

index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP")
| rex field=_raw "VM\s+(?<MachineName>.*)$"
| table _time, PoolId, MachineName, _raw
| transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false
| where eventcount > 1
| search (PoolId="*") (MachineName="*")
| search duration<=700
| stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId
| eval DurationMin = round(DurationMin, 2)
| eval DurationAvg = round(DurationAvg, 2)
| eval DurationMax = round(DurationMax, 2)
| eval ProvDuration = round((EndTime - StartTime), 2)
| eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q")
| eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q")
| table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime
0 Karma

kiran_panchavat
Champion

@NoSpaces 

  • Ensure that both searches (dashboard and manual) are using the same time range.
  • Check the time picker settings in the dashboard. The default time range in a dashboard might be different from the one you used in the search bar.
  • If you have multiple panels, ensure that they are all using the same base search. Sometimes, panels might be referencing different searches, leading to inconsistencies
Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...