Microsoft IIS add-on - no logs being ingested?

[mike_k]

I have a Splunk Standalone instance running at v8.2.10

I have recently installed the Microsoft Add-on for Microsoft IIS (version 1.2.0) on my Splunk server and have also deployed this app to a windows server with IIS installed (and a UF installed). However I seem to be having difficulties getting any logs from this IIS server. 

If I do a search on data in this new index (index=windows_iis), it is returning no results. If I look under Settings>indexes, I can see the newly created index, however it has 0 for event count.

These were the basic steps I have followed so far:

1. I have created a new index for these logs called "windows_iis" - all other settings as default.
2. Installed the Microsoft Add-on for Microsoft IIS on my Splunk Enterprise instance (combined Search Head/Indexer/deployment server).
3. I have copied the contents of this add-on to the /opt/splunk/etc/deployment-apps folder
4. Within the deployment app I have created the following inputs.conf file under the deployment app local directory:
   [monitor://C:\inetpub\logs\LogFiles]
   disabled=false
   sourcetype=ms:iis:auto
   index=windows_iis
5. I have reloaded the deployment server.
6. I have created a new server class and pushed this app out to the IIS server. 

I have gone through and done the following troubleshooting steps:

1. looking on the IIS server in c:\program files\splunkuniversalforwarder\var\log\splunk\splunkd.log, I can see:
   1. UF on IIS server is showing connected to my indexer.
   2. The UF has "adding watch on path: C:\inetpub\logs\LogFiles". So the UF is monitoring the IIS log files.
   3. I am also getting some INFO messages - "ChunkedLBProcessor Failed to find EVENT_BREAKER regex in props.conf for sourcetype: ms:iis:auto. Reverting to the default EVENT_BREAKER regex for now". Not sure how relevant these are? I think my problem might be more fundamental?
2. If I do a search on my Splunk Enterprise instance as follows: "index=_internal host="IIS_Server01" component=Metrics group=per_sourcetype_thruput series="ms:iis:auto" ", I can events being sent from the UF on the IIS server (e.g kbps=0.557, eps=3.3, kb=33, ev=202).
3. I can actually see logs in C:\inetpub\logs\LogFiles\W3SVC1 folder on the IIS server, so there is data there to collect. 

Does the modified local/inputs.conf need to also be configured on the Splunk Enterprise server app or is this inputs.conf configuration only needed on the UF deployment app (which is what I have done)?

Any thoughts on why these events aren't being ingested by my Splunk Enterprise server would be greatly appreciated.

Thanks,

[yeahnah]

Something else to watch out for is the event time extraction is not doing something funny.   You may have already done this type of check, but, if not, try searching over a wider time period window.

Using the tstats command is a quick and efficient way to do this, e.g.

| tstats max(_time) AS _time WHERE sourcetype=ms:iis:auto AND host=IIS_Server01

[mike_k]

@yeahnah 

yeah haven't played around with tstats much really.

If I try this command I don't get any results (even when getting rid of the host portion of the search query).

I'm not really sure what this search is attempting to show me.

[yeahnah · Accepted solution]

The tstats command just searches the time series index files, which are small compared to the raw events and hence quicker to query.

Maybe another obvious question, the role(s) your user belongs to, does have permission to see the index this data is going to.  If you load a file (Add data) via the UI into this index, your user can query it OK?

[mike_k]

Thanks for your tip @yeahnah on manually loading a file into the index and see what it does. This got me started on figuring this out. Did the following:

1. I imported a simple CSV using sourcetype=testsourcetype, index=windows_iis - the result was the data got ingested successfully into the index and i could search on it successfully.
2. I imported a second CSV using sourcetype=ms:iis:auto, index=windows_iis - the result was that as soon as I typed ms:iis:auto into the sourcetype filter and selected this sourcetype, the preview of the data went all weird - kept just the date/time and lost the rest of the data for each event.
3. I imported a third CSV using sourcetype=ms:iis:default:85, index=windows_iis - the result was that the data was ingested successfully into the index and I could search on it successfully.

Not sure what is actually happening when i use sourcetype=ms:iis:auto and why it doesn't work, however I've changed my deployment to use sourcetype=ms:iis:default:85 (which is probably a more appropriate selection anyhow, based on my IIS version).

Thanks for your help. 🙂

[mike_k]

@Tom_Lundie , thanks for your response.

1. Have checked and the index=windows_iis is definitely enabled. Have double checked to make sure that there is no typo's here between name of index and what is in the inputs.conf monitor stanza. The UF is definitely outputting traffic to this index
2. I ran a btool on outputs.conf on the IIS UF, and it is only forwarding traffic out to the one Splunk server. As a part of this work i also enabled the collection of some perfmon stats on this IIS server as well (going into a different directory). These are coming through ok.
3. Where would i find info on routing?
4. If I do a search on my Splunk Enterprise server for: index=_internal source="/opts/splunk/var/log/splunk/splunkd.log" "deleted index", I'm not seeing any results.

Further looking through the metrics log on the IIS server. I can see:

1. group=per_source_thruput, series="C:\inetpub\logs\logfiles\w3svc1\<log_file_name>.log." is showing packets being sent against each of the log files in that directory. So definitely seems to be monitoring the right files.
2. group=per_sourcetype_thruput, series="ms:iis:auto" is showing packets being sent.
3. group=thruput, name=idxsummary, series="windows_iis" is showing packets being sent.
4. group=per_index_thruput, series="windows_iis" is showing packets being sent to the correct index.

If I look at the metrics.log file on my Splunk Enterprise server, I can see that:

1. group=per_index_thruput, series="windows_iis" has non zero eps and ev parameter .... so presumably the splunk enterprise server is seeing this traffic for this index arriving. 
2. group=per_sourcetype_thruput, series="ms:iis:auto" has non zero eps and ev parameter .... so presumably the splunk enterprise server is seeing this traffic for this sourcetype arriving. 

[Tom_Lundie]

Your metrics indicate that your events are arriving, it's peculiar that you can't see them.

@yeahnah calling out _time extraction is a great shout. Definitely explore those ideas. Could your events be older than your frozenTimePeriodInSecs?

Queue routing is defined in transforms.conf and applied via props.conf. I'd check what props.conf apply to your sourcetype, host:: or source:: and make sure that there are no TRANSFORMS calls modifying the queue or index names. TCP routing is defined in inputs.conf but your post has ruled out those issues.

The disabled/deleted index message is an actual warning presented within the Splunk GUI (not logged), look for a blue circle with a number in at the top of the GUI.

Another thing to check is: have you blown your license for today? Seems trivial but its worth checking off.

Finally, does your user have permissions to read that index?

If nothing turns up, I would also suggest a restart of Splunk Enterprise just to rule it out.

[Tom_Lundie]

Your deployment and subsequent troubleshooting steps all look absolutely spot-on so far!

You're right in thinking that you don't need the local inputs.conf on your Splunk Enterprise instance.

I suspect the issue here could be down to a slight misconfiguration or typo.

1. Is the index definitely enabled?
2. Could the UF be forwarding to a different Splunk Instance?
   i.e. Have you got any other outputs.conf servers on your UF?
3. Have you got any routing/index renaming props/transforms applied on Splunk Enterprise?
4. Could this be a typo, on your Splunk Enterprise messages can you see this warning?
   "Received event for unconfigured/disabled/deleted index"
   If so make sure your index name matches your inputs.conf exactly.