I'm having trouble getting splunk to adjust the timezone. The data shows up two hours behind the timezone on the splunk server. I've edited /opt/splunk/etc/system/local/props.conf with the following config:
When i run /opt/splunk/bin/splunk test sourcetype /opt/splunk/bin/splunk test sourcetype my timezone change doesnt show, is that normal? I tried changing the charset variable and that one changed accordingly.
Here's a sample of my log:
2011-05-30T21:59:08+02:00 esxi01 Vpxa: [2011-05-30 21:59:08.742 185A9B90 verbose 'App'] Set internal stats for VM: 16 (vpxa VM id), 80393 (vpxd VM id). Is FT primary? 0
I'm using rsyslog as syslog daemon.
Yes, but in other logs from the same server there is no second timestamp, this is from my vmkernel log from the same server:
2011-05-31T14:06:32+02:00 esxi01: 53:01:47:20.389 cpu6:5034)MigrateNet: vm 5034: 1422: dataSocket 0x4100a2169f00 receive buffer size is 563560
I tried the with the MAX_TIMESTAMP_LOOKAHEAD in my props.conf but when I run splunk test sourcetype it still says Attr:MAX_TIMESTAMP_LOOKAHEAD 45. It seems not all of my changes to the props.conf file are activated. I did a test with charset and that changed to what I specified.
That might be true but the timestamp is correct in regards to the utc timezone on esxi while my splunk server is in timezone CET (UTC+2 summertime and UTC+1 wintertime). All I want is that when I do a search on a specific time or realtime search that I get all the relevant data from that time. As it stands now I have to do special searches for my esxi hosts to see the data in regards to time.
I'm no rsyslog expert, but I've been doing some reading and it looks like, by default, rsyslog ignores incomming timestamps and substitutes its own. This makes me think that either you have a timezone issue on the server running rsyslog or that maybe you want to set the appropriate "...IgnoreMsgTimestamp" directive in rsyslog to off for the socket being used to accept the incomming ESXi logs.
Could it be that the first timestamp is put in by rsyslogd and the second timestamp (the one after "Vpxa: [" is the real timestamp from ESXi?
In that case, you could use
TIME_PREFIX = Vpxa: TZ = UTC
In your props.conf. This will tell it to skip anything that comes before the text, "Vpxa:", thus skipping the first timestamp and using the second one.
The second timestamp is cooler anyway, as it has subsecond resolution. 🙂
NOTE: in this case, you wouldn't need the MAX_TIMESTAMP_LOOKAHEAD parameter.
FYI, The docs for timestamp recognition are here.
It's esxi so it must be rsyslog thats adding it. I have full esx hosts too and there is no problem with thoose. I'll do a test with
MAX_TIMESTAMP_LOOKAHEAD but there must be a "supported" way of solving this?
TZ = <timezone identifier> * The algorithm for determining the time zone for a particular event is as follows: * If the event has a timezone in its raw text (for example, UTC, -08:00), use that. * If TZ is set to a valid timezone string, use that. * Otherwise, use the timezone of the system that is running splunkd. * Defaults to empty.
Because your events are saying they're UTC+2 (note the +02:00 on your timestamp), the timestamp in the event is taking precedence over your TZ setting.
Per VMware KB article 1436, ESXi is always in UTC, so my guess is that either rsyslog is adding the timestamp, or maybe you're using the full ESX instead of ESXi. If you're using ESX, the KB article above will show you how to fix your timezone.
Although fixing your timezone in your source is the preferred fix, you could also try adding
MAX_TIMESTAMP_LOOKAHEAD = 19
To your props.conf so timestamp extraction ignores the timestamp in the event. This is a hack though. Fixing the timestamp on the incomming events is a better deal.