Getting Data In

SplunkStorm & IIS & time stamp

New Member

I've seen a few similar questions asked with answers that either don't apply or don't help, and I apologize in advance if I missed the helpful one somewhere. I'm fairly green on the forwarders so I may be missing something.

I've got the universal forwarder installed on a server and monitoring a single iis log location. I tracked down and am using the inputs.conf file in Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local. It currently has a single entry:
sourcetype = iis
ignoreOlderThan = 1d
followTail = 0
disabled = false

The server happens to live in eastern time, I'm in central and of course IIS logs in UTC. I added an entry in Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf under the [default] stanza "_tzhint = US/Central" so the logs are delivered in my time.

What I end up getting in the RealTime view is logs that are interpreted as "local" time...i.e. a log entry stamped as 2012-10-19 16:39:54 is indexed as 4:39 pm.

Also, I've noticed that the logs are consistently behind by between 3-8 minutes. That is, something logged on the server at say 10:39am doesn't show up in the index until 10:42 (though the index time is correct). At first I thought this was related to the IIS log buffering and flushing, but I can see log entries in the log file quite a while before they make it over.

I've checked the logs and routinely see entries like:
10-19-2012 12:45:17.605 -0400 WARN TcpOutputProc - Raw connection to ip= timed out
10-19-2012 12:45:37.607 -0400 WARN TcpOutputProc - Raw connection to ip= timed out
10-19-2012 12:45:40.123 -0400 INFO TailingProcessor - ...continuing.
10-19-2012 12:45:40.123 -0400 INFO BatchReader - Continuing...
10-19-2012 12:45:50.124 -0400 INFO BatchReader - Could not send data to output queue (parsingQueue), retrying...
10-19-2012 12:45:53.515 -0400 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
10-19-2012 12:46:17.751 -0400 INFO TcpOutputProc - Connected to idx= using ACK.
10-19-2012 12:47:13.831 -0400 INFO TcpOutputProc - Connected to idx= using ACK.

But then I'll see some successes:
10-19-2012 12:37:17.997 -0400 INFO TcpOutputProc - Connected to idx= using ACK.

Thanks in advance.

0 Karma

Splunk Employee
Splunk Employee

Hi Jsajdak.

The IIS logs are always in UTC, and the iis sourcetype in splunk should know it.
So you don't need to add the _tzhint on the input. (but you may need for other inputs that don't have a TZ in the timestamp)

About the delay in the monitoring + forwarding + indexing, 1-3 minutes are in the classic range. If you plan to use realtime searches, please specify a larger time window (like realtime last 15 min).
To verify, please compare the lag between the event timestamp with the index time :

sourcetype=iis | eval lag_sec=_indextime-_time | timechart min(lag_sec) avg(lag_sec) max(lag_sec) by source host

If you see an important lag, it may be :

Your sample from the logs with the parsing queue being blocked, is a symptom of the thruput limit, please follow the guide mentioned above.

0 Karma

Splunk Employee
Splunk Employee

final answer :

On storm you cannot use the classic Splunk timezone settings in props.conf. (they need to be on the indexer, and only have access to the forwarder)
you need to use the special setting _tzhint in inputs.conf


example :

disabled = false

0 Karma

New Member

I'm still not getting the correct time zone. My IIS logs in UTC are still not converting to my project time. I've tried setting up my conf files exactly as it is here: (because Splunk doesn't want to parse out my IIS fields either).

I'm updating the files in Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local

0 Karma

New Member

Thanks for the follow up. The lag times seem to have cleaned themselves up over the weekend. It very well could have been that it was indexing old things (before I had put the ignoreOlderThan switch in).

However, I'm still dealing with the time offset.
I tried uploading an screenshot image but am getting error "upload requires karma>60"..
In any case the IIS log is 2012-10-22 14:16:52
Its being indexed as 10/12/2012 2:16:62 PM. What have I got set wrong?
I don't have any _tzhint setting (took it out Friday).
The project time zone I'm working in is set to US/Central.


0 Karma
Get Updates on the Splunk Community!

Index This | A sphere has three, a circle has two, and a point has zero. What is it?

September 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...