topic Re: Splunk graphs the future in Dashboards & Visualizations

Splunk graphs the future

kwaingrow — Wed, 27 Mar 2013 22:07:23 GMT

I'm running Splunk version: 4.2.3-105575

Searcher and indexers are all set to GMT which may be the cause. Looking at all the events of a particular sourcetype, the Index time, Source time, and even the bar graph Display time are all correct, but the bar graph believes there is 8 hours of data missing in the future. This is affect other graphs we build off and affects our analysis trending. http://www.ugu.com/splunk/gap8hr.jpg

Re: Splunk graphs the future

gfuente — Mon, 01 Apr 2013 09:35:42 GMT

Hello

Have you checked the timezone settings for your user?

Regards

Re: Splunk graphs the future

kwaingrow — Mon, 01 Apr 2013 16:24:41 GMT

Yes. Further investigating reveals.

All splunk servers are running Linux Centos 5.x, all have the same Splunk version: 4.2.3-105575

From the Searcher, the following exhibits the gap between PDT and UTC on our standard OS built indexers (as seen in the attached link above). This we know already know.
sourcetype="servicelog" earliest=-8h splunk_server="indexer022"

We have an indexer on an old non-standard OS that everything works fine and there is no gap.

Also worth noting that the issue does not appear with "some" of the other sourcetypes on indexer022.

An help is greatly appreciated...

Re: Splunk graphs the future

kwaingrow — Mon, 01 Apr 2013 16:25:36 GMT

All other primary splunk conf files (inputs, props, transforms, server, web, etc…) are the same between indexer0001 and indexer022. It also worth noting that the issue does not appear with "some" of the other sourcetypes on indexer022.

Could there be a hidden conf somewhere that has a stanza uniquely configured for this sourcetype (servicelog) and a few others. I haven't found anything in the normal config files.