I have a Splunk Standalone instance running at v8.2.10
I have recently installed the Microsoft Add-on for Microsoft IIS (version 1.2.0) on my Splunk server and have also deployed this app to a windows server with IIS installed (and a UF installed). However I seem to be having difficulties getting any logs from this IIS server.
If I do a search on data in this new index (index=windows_iis), it is returning no results. If I look under Settings>indexes, I can see the newly created index, however it has 0 for event count.
These were the basic steps I have followed so far:
I have gone through and done the following troubleshooting steps:
Does the modified local/inputs.conf need to also be configured on the Splunk Enterprise server app or is this inputs.conf configuration only needed on the UF deployment app (which is what I have done)?
Any thoughts on why these events aren't being ingested by my Splunk Enterprise server would be greatly appreciated.
Thanks,
The tstats command just searches the time series index files, which are small compared to the raw events and hence quicker to query.
Maybe another obvious question, the role(s) your user belongs to, does have permission to see the index this data is going to. If you load a file (Add data) via the UI into this index, your user can query it OK?
Something else to watch out for is the event time extraction is not doing something funny. You may have already done this type of check, but, if not, try searching over a wider time period window.
Using the tstats command is a quick and efficient way to do this, e.g.
| tstats max(_time) AS _time WHERE sourcetype=ms:iis:auto AND host=IIS_Server01
yeah haven't played around with tstats much really.
If I try this command I don't get any results (even when getting rid of the host portion of the search query).
I'm not really sure what this search is attempting to show me.
The tstats command just searches the time series index files, which are small compared to the raw events and hence quicker to query.
Maybe another obvious question, the role(s) your user belongs to, does have permission to see the index this data is going to. If you load a file (Add data) via the UI into this index, your user can query it OK?
Thanks for your tip @yeahnah on manually loading a file into the index and see what it does. This got me started on figuring this out. Did the following:
Not sure what is actually happening when i use sourcetype=ms:iis:auto and why it doesn't work, however I've changed my deployment to use sourcetype=ms:iis:default:85 (which is probably a more appropriate selection anyhow, based on my IIS version).
Thanks for your help. 🙂
@Tom_Lundie , thanks for your response.
Further looking through the metrics log on the IIS server. I can see:
If I look at the metrics.log file on my Splunk Enterprise server, I can see that:
Your metrics indicate that your events are arriving, it's peculiar that you can't see them.
@yeahnah calling out _time extraction is a great shout. Definitely explore those ideas. Could your events be older than your frozenTimePeriodInSecs?
Queue routing is defined in transforms.conf and applied via props.conf. I'd check what props.conf apply to your sourcetype, host:: or source:: and make sure that there are no TRANSFORMS calls modifying the queue or index names. TCP routing is defined in inputs.conf but your post has ruled out those issues.
The disabled/deleted index message is an actual warning presented within the Splunk GUI (not logged), look for a blue circle with a number in at the top of the GUI.
Another thing to check is: have you blown your license for today? Seems trivial but its worth checking off.
Finally, does your user have permissions to read that index?
If nothing turns up, I would also suggest a restart of Splunk Enterprise just to rule it out.
Your deployment and subsequent troubleshooting steps all look absolutely spot-on so far!
You're right in thinking that you don't need the local inputs.conf on your Splunk Enterprise instance.
I suspect the issue here could be down to a slight misconfiguration or typo.