:edit: summary of results
Problem was system timezone coupled to Splunk's use of NOW in default queries. Either fix your Timezone settings or use Advanced in the Search App's time dropdown and specify something like latest=+2d.
See also docs.splunk.com/Documentation/Splunk/7.3.0/Search/Specifytimemodifiersinyoursearch
----- better problem description (if I knew this in advance I wouldn't have needed to ask) -----
I have records that appear to occur in the future due to a timezone conflict. Splunk is refusing to show these records.
------ original (and decidedly poor) problem description -----
Newbie...
After running a search in the default Searching & Reporting App, my searches stop returning any new records.
I've tried logout/login, reentering my search but until something like the next day, I can't see any new data.
What am I doing wrong?
PS: I know Splunk is ingesting the new data because I can run the same search using the API and see all the new records.
When you run a search from the GUI, there is a Timepicker
that will default to some value that is totally obvious and you can change it to whatever you like. However when you run it from the API (or from the CLI), there is no timepicker and who knows what it is defaulting to. You can ABSOLUTELY make both modes give the same results by adding SPL time specifiers like earliest=-2d latest=now
.
That is the first part of the question, now that you have both modes returning the same results (because they are now actually running the same search), why are you not getting all events that you were expecting. Almost always it is because you are letting splunk guess where the timestamps are and it is guessing wrong. In most cases like yours where you KNOW there is data but cannot find it, it is because the TZ
is wrong and the events are being thrown into the future. The only Timepicker
value that shows these is All time
and I have heard that in the very latest releases, this has been modified so that the latest
value in that is no longer Infinity+
but now
so you will have to go to the Advanced
area (or SPL) and use something like earliest=-2d latest=@d+1y
. The meta woot
app can help you sort through these timestamping problems but often it is quite complex and people resort to PS from companies like ours to get it done completely/well/right.
When you run a search from the GUI, there is a Timepicker
that will default to some value that is totally obvious and you can change it to whatever you like. However when you run it from the API (or from the CLI), there is no timepicker and who knows what it is defaulting to. You can ABSOLUTELY make both modes give the same results by adding SPL time specifiers like earliest=-2d latest=now
.
That is the first part of the question, now that you have both modes returning the same results (because they are now actually running the same search), why are you not getting all events that you were expecting. Almost always it is because you are letting splunk guess where the timestamps are and it is guessing wrong. In most cases like yours where you KNOW there is data but cannot find it, it is because the TZ
is wrong and the events are being thrown into the future. The only Timepicker
value that shows these is All time
and I have heard that in the very latest releases, this has been modified so that the latest
value in that is no longer Infinity+
but now
so you will have to go to the Advanced
area (or SPL) and use something like earliest=-2d latest=@d+1y
. The meta woot
app can help you sort through these timestamping problems but often it is quite complex and people resort to PS from companies like ours to get it done completely/well/right.
Thanks. So my problem comes down to the implicit presence of -now- in the timepicker and that even real-time includes -now- as the latest and that my system timezone is probably GMT.
Thanks also for the @d+1yr syntax. It will help me work around the issue until I get a consistent TZ configuration.
Give is SOMETHING to dig into. What version of Splunk? What OS? What architecture structure? What SPL are you running? What in the world do you EXACTLY mean by using the API
?
OK. In more detail.
splunk-7.3.0-657388c7a488.x86_64
Running on Centos 7
The query I'm running in the default Splunk Search GUI is:
source="/var/log/splunk-test.log" index="testing" sourcetype="_json" | where systemStatus=1
The most recent record returned by this query is:
{"systemStatus": 1, "device": "Core 3", "temp": "34.0", "message": "", "time": "2019-08-07T*06*:01:08.329579"}
When I run the same query via the Python splunk-sdk 1.6.6 API wrapper I see many more recent records including this one:
{"systemStatus": 1, "device": "Core 3", "temp": "36.0", "message": "", "time": "2019-08-07T*23*:39:50.879879"}
Other than this odd temporal behaviour the query works properly in both environments