Splunk graphs the future

[kwaingrow]

I'm running Splunk version: 4.2.3-105575

Searcher and indexers are all set to GMT which may be the cause. Looking at all the events of a particular sourcetype, the Index time, Source time, and even the bar graph Display time are all correct, but the bar graph believes there is 8 hours of data missing in the future. This is affect other graphs we build off and affects our analysis trending. http://www.ugu.com/splunk/gap8hr.jpg

[gfuente]

Hello

Have you checked the timezone settings for your user?

Regards

[kwaingrow]

All other primary splunk conf files (inputs, props, transforms, server, web, etc…) are the same between indexer0001 and indexer022. It also worth noting that the issue does not appear with "some" of the other sourcetypes on indexer022.

Could there be a hidden conf somewhere that has a stanza uniquely configured for this sourcetype (servicelog) and a few others. I haven't found anything in the normal config files.

[kwaingrow]

Yes. Further investigating reveals.

All splunk servers are running Linux Centos 5.x, all have the same Splunk version: 4.2.3-105575

From the Searcher, the following exhibits the gap between PDT and UTC on our standard OS built indexers (as seen in the attached link above). This we know already know.
sourcetype="servicelog" earliest=-8h splunk_server="indexer022"

We have an indexer on an old non-standard OS that everything works fine and there is no gap.

Also worth noting that the issue does not appear with "some" of the other sourcetypes on indexer022.

An help is greatly appreciated...