Archive

Why does Search App seem to get stuck in the past?

lfast12
Explorer

:edit: summary of results
Problem was system timezone coupled to Splunk's use of NOW in default queries. Either fix your Timezone settings or use Advanced in the Search App's time dropdown and specify something like latest=+2d.
See also docs.splunk.com/Documentation/Splunk/7.3.0/Search/Specifytimemodifiersinyoursearch

----- better problem description (if I knew this in advance I wouldn't have needed to ask) -----
I have records that appear to occur in the future due to a timezone conflict. Splunk is refusing to show these records.

------ original (and decidedly poor) problem description -----
Newbie...
After running a search in the default Searching & Reporting App, my searches stop returning any new records.

I've tried logout/login, reentering my search but until something like the next day, I can't see any new data.

What am I doing wrong?

PS: I know Splunk is ingesting the new data because I can run the same search using the API and see all the new records.

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

When you run a search from the GUI, there is a Timepicker that will default to some value that is totally obvious and you can change it to whatever you like. However when you run it from the API (or from the CLI), there is no timepicker and who knows what it is defaulting to. You can ABSOLUTELY make both modes give the same results by adding SPL time specifiers like earliest=-2d latest=now.

That is the first part of the question, now that you have both modes returning the same results (because they are now actually running the same search), why are you not getting all events that you were expecting. Almost always it is because you are letting splunk guess where the timestamps are and it is guessing wrong. In most cases like yours where you KNOW there is data but cannot find it, it is because the TZ is wrong and the events are being thrown into the future. The only Timepicker value that shows these is All time and I have heard that in the very latest releases, this has been modified so that the latest value in that is no longer Infinity+ but now so you will have to go to the Advanced area (or SPL) and use something like earliest=-2d latest=@d+1y. The meta woot app can help you sort through these timestamping problems but often it is quite complex and people resort to PS from companies like ours to get it done completely/well/right.

View solution in original post

woodcock
Esteemed Legend

When you run a search from the GUI, there is a Timepicker that will default to some value that is totally obvious and you can change it to whatever you like. However when you run it from the API (or from the CLI), there is no timepicker and who knows what it is defaulting to. You can ABSOLUTELY make both modes give the same results by adding SPL time specifiers like earliest=-2d latest=now.

That is the first part of the question, now that you have both modes returning the same results (because they are now actually running the same search), why are you not getting all events that you were expecting. Almost always it is because you are letting splunk guess where the timestamps are and it is guessing wrong. In most cases like yours where you KNOW there is data but cannot find it, it is because the TZ is wrong and the events are being thrown into the future. The only Timepicker value that shows these is All time and I have heard that in the very latest releases, this has been modified so that the latest value in that is no longer Infinity+ but now so you will have to go to the Advanced area (or SPL) and use something like earliest=-2d latest=@d+1y. The meta woot app can help you sort through these timestamping problems but often it is quite complex and people resort to PS from companies like ours to get it done completely/well/right.

View solution in original post

lfast12
Explorer

Thanks. So my problem comes down to the implicit presence of -now- in the timepicker and that even real-time includes -now- as the latest and that my system timezone is probably GMT.

Thanks also for the @d+1yr syntax. It will help me work around the issue until I get a consistent TZ configuration.

0 Karma

woodcock
Esteemed Legend

Give is SOMETHING to dig into. What version of Splunk? What OS? What architecture structure? What SPL are you running? What in the world do you EXACTLY mean by using the API?

0 Karma

lfast12
Explorer

OK. In more detail.
splunk-7.3.0-657388c7a488.x86_64
Running on Centos 7

The query I'm running in the default Splunk Search GUI is:
source="/var/log/splunk-test.log" index="testing" sourcetype="_json" | where systemStatus=1

The most recent record returned by this query is:
{"systemStatus": 1, "device": "Core 3", "temp": "34.0", "message": "", "time": "2019-08-07T*06*:01:08.329579"}

When I run the same query via the Python splunk-sdk 1.6.6 API wrapper I see many more recent records including this one:
{"systemStatus": 1, "device": "Core 3", "temp": "36.0", "message": "", "time": "2019-08-07T*23*:39:50.879879"}

Other than this odd temporal behaviour the query works properly in both environments

0 Karma