Getting Data In

Wrong time offset in indexed data

let_eat_bee
New Member

Hello. There is a few cisco routers are sending syslogs via UDP to splunk server. Earlier everything was ok, but recently I noticed that there is an offset in timestamps despite time settings on splunk server are absolutely correct. This is how it looks:
alt text
I tried to fix that by configuration of props.conf such as
TZ = GMT+3 or
TZ = GMT
if I clear any timezone config from props.conf that the shift becomes 2 hours...
My actual timezone is GMT+3. And the time on a server is correct.
also i noticed that search command

* | stats count AS tnow | eval tnow = now() | convert ctime(tnow)

shows the correct time as it is on the server.

can anybody explain how to workaroung this problem and what the cause of it ?

Tags (2)
0 Karma

woodcock
Esteemed Legend

I really cannot follow what you have said well enough to give you a full answer but I can tell you a few things that will allow you to help yourself. The main search you need is this one (assuming that your forwarder pipeline has near-zero latency, which is usually the case):

index = * | eval lagSecs = _indextime - _time | stats avg(lagSecs) by index,host,sourcetype

The avg should NEVER be negative and should generally be in the low-hundreds or smaller. Assuming you have correct clock time and no drift (NTP in place), this search will show you which hosts for which sourcetypes need the TZ value to be adjusted but there is another thing to consider: if you have overridden your host or sourcetype, you must use the original/non-overridden value for your stanza header. Many times, this is not practical and the best (only?) other option is to have each host write to his own private subdirectory and then use a source-based stanza (which is now an analog for host) in props.conf to set the TZ. If you are using DST, make sure that you do NOT use the GMT*-based TZs but use the US/*-based TZs.

let_eat_bee
New Member

My splunk server and the routers are located in a single time zone with properly confirured time.
It is and I am are located in Europe/Minsk timezone (Belarus).
This month is the last for our country in UTC+2 DST timezone. After Oct 30 timezone becomes UTC+3 without changing time and without DST in future.

0 Karma

kristian_kolb
Ultra Champion

Hi, could you clarify a little more? Are you and the routers in different timezones? Where are you located? , If Daylight Saving Time is observed in your country, have you switched to Winter Time recently?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...