How best to deal with servers changing date/time during DST testing

[mikelanghorst]

Twice a year we have a set of servers used for testing our apps during DST time changes. About 2 months before we will set the time a couple of days before the time change, let the apps roll thru it then reset the time to before the change. Repeat many times. Trying to keep all the events straight could get confusing. Having the users search by index time, or forcing those events to be set to index time could be difficult as well. Typically we'll offset by 12hrs so that 2am is 2pm, but it's not always the case.

What would you recommend to keep events from these servers straight? I'm thinking maybe just set those hosts to put everything in a different index and clean it, but doesn't seem ideal either.

[ramonej]

An easier option for your DST testing (instead of managing a temp index, cleaning data, & changing all the server times back/forth), would be a 3rd party tool to manage the entire time change. It especially helps when the time change spans multiple components. I've had success with TimeShiftX by the company Vornex

[mikelanghorst]

Best for this issue is to just create a temp index and clean the data during the date rollbacks.

[gkanapathy]

Well, the easiest solution is to log everything in UTC. There is only one time recorded, and it is not affected by UTC. Then the only questions are how to display it, and how to interpret input when a user provides a time range, which is usually easier and more flexible. Generally display is done in local time using rules for the local zone, and input query ranges usually follow the same rule. This almost always matches user expectations.

The next easiest solution is to make sure your logs record the time zone as part of the timestamp. Splunk can parse this and convert it to UTC and everything will work fine.

If the applications don't record correct times on a DST switch, or don't record the time zone (or worse, as I have seen, record the time zone incorrectly), there's not a lot we can do, as it's not possible to simply look at a timestamp and know what the actual time was.

[mikelanghorst]

Yea, our use case here is pretty unique I think. Thanks for the help, just needed to verify that I wasn't missing something. DATETIME_CONFIG=current would cause confusion as well due to the hours being offset as well.

Temp index it is.

[gkanapathy]

Uh this sounds like a special-case test specifically of DST. I suppose either you send to a special index, or you set DATETIME_CONFIG = current and ignore the timestamps in the data.

[mikelanghorst]

The issue I really have here isn't the actual switch during a single DST change. The problem is dealing with a set of servers that might create logs for a 3 day period 10,15, maybe 20 times. We actually set the date/time of the servers for the Thursday before the time change, setup our data and let it run thru Sunday morning (while it's actually Mon-Thursday in September). Once we reach Sunday, the dates on the servers are reset to that Thursday and the whole process started over.

Any ideas other than using a temp index?