IIS logs and timezone

[naydenk]

Hello
I have a problem with IIS logs' timestamps (the common issue where the events are indexed as they are logged, in GMT, and show up 4-5 hours in the future, since I am in the US). I searched splunk-base answers and I see a lot of people asking this same question. Most of the answers are the same, although there are variations. I have tried a lot of different options, the common and not so common solutions and nothing seems to work for me. Here is my current setup:

• Universal Forwarder consuming IIS logs
• The UF is forwarding its data to an intermediary splunk instance, which in turn sends them to an indexer.
• I have created an deployment app called index_SSA that the client gets which contains props and inputs.conf files where I am trying to set the TZ properties. The directory of the app, where I have the conf files is "C:\Program Files\SplunkUniversalForwarder\etc\apps\index_SSA\default"

The inputs.conf file:

[monitor://C:\inetpub\logs]
disabled = false
followTail = 0
index = SSA

The props.conf file (currently):

.....

[source::(?i)...\inetpub\Logs\W3SVC2\ex(.\d+)?.log]

sourcetype = iisw3c

TZ = GMT

........

I have also tried this in the props.conf:

.......

[iisw3c]

sourcetype = iisw3c

TZ = GMT

......

So, obviously I am doing something wrong, because none of the variations I have tried have worked... Ideally, I would like to be able to index IIS logs and their headers, be able to search and have the search reflect the correct time (i.e. when I search for events logged in the last 15 min, I want to see them in the search, and not have to search for events 4-5 hours in the future). I also would like Splunk to recognize and extract the fields from the IIS logs, so i can search for those values (or if I have to do the field extraction manually, then that is fine, but I want to know if I should do that vs. expect Splunk to automatically do it)

If anyone can provide the stanzas and values that I should use, so I don't lose my mind...

Thanks in advance!

[mwhite_splunk]

I think you may be overcomplicating the situation. This should work:

[iisw3c]

TZ = GMT

and it needs to be in:

$SPLUNK_HOME/etc/system/local/

and

$SPLUNK_HOME/etc/apps/relevant_app/

on your UF.

[naydenk]

... continued ...

When I search, I expect events logged in the from the past 10 min to show up in the "Last 15 min". In addition, I would like to see the IIS fields recognized (sc-status, uri-query, c-ip, etc.), which currently are not showing up either - not sure if this is a symptom related to this issue, or something else I need to deal with later.

Thanks again for your patience and help... 🙂

[naydenk]

Well, I agree, it is a mismatch and I think I am making the changes in the right place, but it is not being reflected when I do the searches, so I am either making the changes in the wrong config, or there is something else that is overriding it. Just to sum up:

All splunk instances (UF, intermediary, indexer/search head) are in EST. The IIS log file's time stamps are in UTC. I have the changes I showed above in the props.conf and inputs.conf of the UF where the logs are being consumed (IIS server).

... will continue in the next comment ...

[mwhite_splunk]

You might have a TZ mismatch issue. If I'm not mistaken, Splunk assumes everything comes in UTC, then you have to tell it what to display. Gerald Kanapathy has a much better explanation of it here:

http://splunk-base.splunk.com/answers/12014/utc-to-local-time-zone-conversion-on-the-fly-splunk-for-...

[naydenk]

Thanks mwhite_splunk. Here is what I have now:

"$SPLUNK_HOME\etc\system\local\props.conf"

[iisw3c]
TZ = GMT

"$SPLUNK_HOME\etc\apps\index_SSA\default\props.conf"

[iisw3c]
TZ = GMT

[source::(?i)...\inetpub\Logs\W3SVC2\ex(.\d+)?.log]
sourcetype = iisw3c

(plus some more source:: stanzas for other log dirs)

"$SPLUNK_HOME\etc\apps\index_OSM\default\inputs.conf"

[monitor://C:\inetpub\logs]
disabled = false
followTail = 0
sourcetype = iisw3c
index = testindex

(plus some more monitor: stanzas for other log dirs)

A new file got indexed, but still the wrong time shows on the timeline.

[naydenk]

Yes, I restart the client UF (where the indexed logs reside) after every change. Originally I was making the change to the app on the DS, then waiting for it to propagate to the client, but after a few tests I got impatient and was just changing the props.conf and inputs.conf directly on the client, then restarting the service.

Regarding the logs - after every change I drop a new log, with new date stamps in the C:\inetpub\logs directory so that it is "fresh" data. I see the data indexed in the splunk search app, but it is with the original time stamps, 4 hours in the future...

[Takajian]

We will need to restart splunk in order to reflect configuration changed by CLI. Did you restart splunk after changing props.conf? And the configuration affect new coming data, not indexed data.