GMT IIS logs won't display real-time

[wrangler2x]

I am taking IIS logs from an Exchange server, which are in GMT. In the indexer's props.conf I have 'TZ = GMT'. When I type in 'index=exchange_index' on the search bar with the time-frame of 'Today' I see the latest logs (with the current time in my timezone -- pst8pdt) at the top of the search.

However, if I change the time-frame to Real-time, no log entries show up. It does not matter what the window is, 30 second, 1 minute, 5 minute, 30 minute, 1 hour... nothing shows up. Why would this be? I don't have that problem on any of my other indexed data. But then, it is all logged in the local time zone. Just the iis logs are GMT.

[wrangler2x]

Here is what I have discovered. The splunk forwarder was parsing the data, and assigning a local time zone to the data. When this happens, there is nothing you can do to change the time zone on the indexer. Once I figured this out, I simply put 'TZ = GMT' in the props.conf for the forwarder. Then logs displayed in real time.

Something interesting I discovered along the way is that--no matter what time zone the logs are from--splunk stores them in GMT, along with the time zone so they can be displayed in search results as the time local to the search head. And when you search based on time, it is not the match on the search data's time, but on the local time that splunk displays to the left of the displayed log entry.

[alacercogitatus]

I found that with IIS, I had to place the stanza in etc/system/local/props.conf on each indexer, not just in an app on the indexer. Can you verify that the time in the raw text is the correct offset from the time in the UI?

[wrangler2x]

Also, I started wondering if maybe the systems were off in time. So I did a search which had just the index name and 'earliest=-1s'. The second I hit return on another window pane I hit enter also, which ran the unix date command on that system. It is the system I run the indexer on. The time of the top record in the Splunk GUI was 1 second off of the time on the date command on the indexer system. So the two systems are either exactly or at most 1 second apart.

[wrangler2x]

I do have the stanza in etc/system/local/props.conf.

In order to answer your question, I used custom time to display records from a 1 second snapshot at 00:00:00 in the GUI, then went to create report, then did an export as .csv. In the opened excel spreadsheet, I used epoch time conversion =(A2/ 86400) + 25569 and formatted that column for date and time to the second. The result was 3/25/2013 07:00:000, which is exactly right for iis records at midnight our time, because they are GMT.