Getting Data In

IIS logs and timezone

naydenk
Path Finder

Hello
I have a problem with IIS logs' timestamps (the common issue where the events are indexed as they are logged, in GMT, and show up 4-5 hours in the future, since I am in the US). I searched splunk-base answers and I see a lot of people asking this same question. Most of the answers are the same, although there are variations. I have tried a lot of different options, the common and not so common solutions and nothing seems to work for me. Here is my current setup:

  • Universal Forwarder consuming IIS logs
  • The UF is forwarding its data to an intermediary splunk instance, which in turn sends them to an indexer.
  • I have created an deployment app called index_SSA that the client gets which contains props and inputs.conf files where I am trying to set the TZ properties. The directory of the app, where I have the conf files is "C:\Program Files\SplunkUniversalForwarder\etc\apps\index_SSA\default"

The inputs.conf file:

[monitor://C:\inetpub\logs]
disabled = false
followTail = 0
index = SSA

The props.conf file (currently):

.....

[source::(?i)...\inetpub\Logs\W3SVC2\ex(.\d+)?.log]

sourcetype = iisw3c

TZ = GMT

........

I have also tried this in the props.conf:

.......

[iisw3c]

sourcetype = iisw3c

TZ = GMT

......

So, obviously I am doing something wrong, because none of the variations I have tried have worked... Ideally, I would like to be able to index IIS logs and their headers, be able to search and have the search reflect the correct time (i.e. when I search for events logged in the last 15 min, I want to see them in the search, and not have to search for events 4-5 hours in the future). I also would like Splunk to recognize and extract the fields from the IIS logs, so i can search for those values (or if I have to do the field extraction manually, then that is fine, but I want to know if I should do that vs. expect Splunk to automatically do it)

If anyone can provide the stanzas and values that I should use, so I don't lose my mind...

Thanks in advance!

Tags (2)

mwhite_splunk
Splunk Employee
Splunk Employee

I think you may be overcomplicating the situation. This should work:

[iisw3c]

TZ = GMT

and it needs to be in:

$SPLUNK_HOME/etc/system/local/

and

$SPLUNK_HOME/etc/apps/relevant_app/

on your UF.

0 Karma

naydenk
Path Finder

... continued ...

When I search, I expect events logged in the from the past 10 min to show up in the "Last 15 min". In addition, I would like to see the IIS fields recognized (sc-status, uri-query, c-ip, etc.), which currently are not showing up either - not sure if this is a symptom related to this issue, or something else I need to deal with later.

Thanks again for your patience and help... 🙂

0 Karma

naydenk
Path Finder

Well, I agree, it is a mismatch and I think I am making the changes in the right place, but it is not being reflected when I do the searches, so I am either making the changes in the wrong config, or there is something else that is overriding it. Just to sum up:

All splunk instances (UF, intermediary, indexer/search head) are in EST. The IIS log file's time stamps are in UTC. I have the changes I showed above in the props.conf and inputs.conf of the UF where the logs are being consumed (IIS server).

... will continue in the next comment ...

0 Karma

mwhite_splunk
Splunk Employee
Splunk Employee

You might have a TZ mismatch issue. If I'm not mistaken, Splunk assumes everything comes in UTC, then you have to tell it what to display. Gerald Kanapathy has a much better explanation of it here:

http://splunk-base.splunk.com/answers/12014/utc-to-local-time-zone-conversion-on-the-fly-splunk-for-...

0 Karma

naydenk
Path Finder

Thanks mwhite_splunk. Here is what I have now:

"$SPLUNK_HOME\etc\system\local\props.conf"

[iisw3c]
TZ = GMT

"$SPLUNK_HOME\etc\apps\index_SSA\default\props.conf"

[iisw3c]
TZ = GMT

[source::(?i)...\inetpub\Logs\W3SVC2\ex(.\d+)?.log]
sourcetype = iisw3c

(plus some more source:: stanzas for other log dirs)

"$SPLUNK_HOME\etc\apps\index_OSM\default\inputs.conf"

[monitor://C:\inetpub\logs]
disabled = false
followTail = 0
sourcetype = iisw3c
index = testindex

(plus some more monitor: stanzas for other log dirs)

A new file got indexed, but still the wrong time shows on the timeline.

0 Karma

naydenk
Path Finder

Yes, I restart the client UF (where the indexed logs reside) after every change. Originally I was making the change to the app on the DS, then waiting for it to propagate to the client, but after a few tests I got impatient and was just changing the props.conf and inputs.conf directly on the client, then restarting the service.

Regarding the logs - after every change I drop a new log, with new date stamps in the C:\inetpub\logs directory so that it is "fresh" data. I see the data indexed in the splunk search app, but it is with the original time stamps, 4 hours in the future...

0 Karma

Takajian
Builder

We will need to restart splunk in order to reflect configuration changed by CLI. Did you restart splunk after changing props.conf? And the configuration affect new coming data, not indexed data.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...