SplunkStorm & IIS & time stamp

[jsajdak]

I've seen a few similar questions asked with answers that either don't apply or don't help, and I apologize in advance if I missed the helpful one somewhere. I'm fairly green on the forwarders so I may be missing something.

I've got the universal forwarder installed on a server and monitoring a single iis log location. I tracked down and am using the inputs.conf file in Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local. It currently has a single entry:
[monitor://M:\web_logs\site_directory*.log]
sourcetype = iis
ignoreOlderThan = 1d
followTail = 0
disabled = false

The server happens to live in eastern time, I'm in central and of course IIS logs in UTC. I added an entry in Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf under the [default] stanza "_tzhint = US/Central" so the logs are delivered in my time.

What I end up getting in the RealTime view is logs that are interpreted as "local" time...i.e. a log entry stamped as 2012-10-19 16:39:54 is indexed as 4:39 pm.

Also, I've noticed that the logs are consistently behind by between 3-8 minutes. That is, something logged on the server at say 10:39am doesn't show up in the index until 10:42 (though the index time is correct). At first I thought this was related to the IIS log buffering and flushing, but I can see log entries in the log file quite a while before they make it over.

I've checked the logs and routinely see entries like:
10-19-2012 12:45:17.605 -0400 WARN TcpOutputProc - Raw connection to ip=184.73.47.206:9997 timed out
10-19-2012 12:45:37.607 -0400 WARN TcpOutputProc - Raw connection to ip=184.73.47.206:9997 timed out
10-19-2012 12:45:40.123 -0400 INFO TailingProcessor - ...continuing.
10-19-2012 12:45:40.123 -0400 INFO BatchReader - Continuing...
10-19-2012 12:45:50.124 -0400 INFO BatchReader - Could not send data to output queue (parsingQueue), retrying...
10-19-2012 12:45:53.515 -0400 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
10-19-2012 12:46:17.751 -0400 INFO TcpOutputProc - Connected to idx=67.202.7.237:9997 using ACK.
10-19-2012 12:47:13.831 -0400 INFO TcpOutputProc - Connected to idx=67.202.7.237:9997 using ACK.

But then I'll see some successes:
10-19-2012 12:37:17.997 -0400 INFO TcpOutputProc - Connected to idx=184.73.47.206:9997 using ACK.

Thanks in advance.

[yannK]

Hi Jsajdak.

The IIS logs are always in UTC, and the iis sourcetype in splunk should know it.
So you don't need to add the _tzhint on the input. (but you may need for other inputs that don't have a TZ in the timestamp)

About the delay in the monitoring + forwarding + indexing, 1-3 minutes are in the classic range. If you plan to use realtime searches, please specify a larger time window (like realtime last 15 min).
To verify, please compare the lag between the event timestamp with the index time :

sourcetype=iis | eval lag_sec=_indextime-_time | timechart min(lag_sec) avg(lag_sec) max(lag_sec) by source host

If you see an important lag, it may be :

• historical logs were indexed (with a very old timestamp)
• they are congestion in the forwarder, please check the thruput limit, http://docs.splunk.com/Documentation/Storm/Storm/User/Setupauniversalforwarderonnix#Remove_the_defau...
• network latency
• busy storm indexers

Your sample from the logs with the parsing queue being blocked, is a symptom of the thruput limit, please follow the guide mentioned above.

[yannK]

final answer :

On storm you cannot use the classic Splunk timezone settings in props.conf. (they need to be on the indexer, and only have access to the forwarder)
you need to use the special setting _tzhint in inputs.conf

see http://docs.splunk.com/Documentation/Storm/latest/User/SetupauniversalforwarderonWindows#Change_the_...

example :

[monitor://C:\path\to\my\iis\logs\]
disabled = false
sourcetype=iis
_tzhint=GMT


[jsajdak]

I'm still not getting the correct time zone. My IIS logs in UTC are still not converting to my project time. I've tried setting up my conf files exactly as it is here: (because Splunk doesn't want to parse out my IIS fields either).

http://splunk-base.splunk.com/answers/24986/iis-log-fields-not-parsing

I'm updating the files in Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local
inputs.conf
props.conf
transforms.conf

[jsajdak]

Thanks for the follow up. The lag times seem to have cleaned themselves up over the weekend. It very well could have been that it was indexing old things (before I had put the ignoreOlderThan switch in).

However, I'm still dealing with the time offset.
I tried uploading an screenshot image but am getting error "upload requires karma>60"..
In any case the IIS log is 2012-10-22 14:16:52
Its being indexed as 10/12/2012 2:16:62 PM. What have I got set wrong?
I don't have any _tzhint setting (took it out Friday).
The project time zone I'm working in is set to US/Central.

Jason