I have a Splunk Standalone instance running at v8.2.10 I have recently installed the Microsoft Add-on for Microsoft IIS (version 1.2.0) on my Splunk server and have also deployed this app to a windows server with IIS installed (and a UF installed). However I seem to be having difficulties getting any logs from this IIS server.  If I do a search on data in this new index (index=windows_iis), it is returning no results. If I look under Settings>indexes, I can see the newly created index, however it has 0 for event count.   These were the basic steps I have followed so far: I have created a new index for these logs called "windows_iis" - all other settings as default. Installed the Microsoft Add-on for Microsoft IIS on my Splunk Enterprise instance (combined Search Head/Indexer/deployment server). I have copied the contents of this add-on to the /opt/splunk/etc/deployment-apps folder Within the deployment app I have created the following inputs.conf file under the deployment app local directory: [monitor://C:\inetpub\logs\LogFiles] disabled=false sourcetype=ms:iis:auto index=windows_iis I have reloaded the deployment server. I have created a new server class and pushed this app out to the IIS server.  I have gone through and done the following troubleshooting steps: looking on the IIS server in c:\program files\splunkuniversalforwarder\var\log\splunk\splunkd.log, I can see: UF on IIS server is showing connected to my indexer. The UF has "adding watch on path: C:\inetpub\logs\LogFiles". So the UF is monitoring the IIS log files. I am also getting some INFO messages - "ChunkedLBProcessor Failed to find EVENT_BREAKER regex in props.conf for sourcetype: ms:iis:auto. Reverting to the default EVENT_BREAKER regex for now". Not sure how relevant these are? I think my problem might be more fundamental? If I do a search on my Splunk Enterprise instance as follows: "index=_internal host="IIS_Server01" component=Metrics group=per_sourcetype_thruput series="ms:iis:auto" ", I can events being sent from the UF on the IIS server (e.g kbps=0.557, eps=3.3, kb=33, ev=202). I can actually see logs in C:\inetpub\logs\LogFiles\W3SVC1 folder on the IIS server, so there is data there to collect.  Does the modified local/inputs.conf need to also be configured on the Splunk Enterprise server app or is this inputs.conf configuration only needed on the UF deployment app (which is what I have done)? Any thoughts on why these events aren't being ingested by my Splunk Enterprise server would be greatly appreciated. Thanks, ... View more

I have a standalone instance of Splunk. I am running both: Splunk Add-on for Unix and Linux, and Splunk App for Unix. Since the Splunk App for Unix has reached End-of-Life and is not required in my deployment anymore i am looking to remove it. Initially i tried just using Splunk command: ./splunk remove app splunk_app_for_nix However noticed that this impacts the index "os" used by the Splunk Add-on for Unix and Linux. The index no longer appears in the web gui under settings>indexes. If i look in the CLI, i can still see data in /opt/splunk/os/db, so the data still appears to be there, but is not being used apparently.... I am getting Message saying "Received event for unconfigured/disabled/deleted index=os ...", so am not entirely sure what the status of this index is now. What is the best way to remove this app without affecting the index? Thanks,     ... View more

I am currently running an on-Prem Splunk installation and am trying to figure out the best approach for ingesting data from our VMware environment. Currently i've got just a very basic setup with syslogs from my ESXi hosts and vCenter going to a Syslog server and monitored by a universal forwarder on the syslog server for forwarding to my indexers. That all works reasonably well, however would like to be able to use some pre-developed dashboards. I have noted that there are a number of VMware add-ons that can be used. Also we are looking to move towards ITSI in the near future so would like to amend the way i ingest WMware data so that it is compatible with this. At a fundamental level, I think i'm struggling with understanding the difference between "Splunk add-on for VMware" and "Splunk add-on for VMware metrics". Is there a reason why i would pick one of these over the other (or do you normally install both)? ... View more

Hi, I am running a single instance Splunk deployment on Linux and am planning on upgrading a bunch of Apps on my Splunk Enterprise server (there are about 7 that need upgrading ... a mix of Apps and Add-ons) . I was intending to upgrade the Apps using the GUI. My question is whether it is better to restart when prompted by Splunk (potentially after each app is upgraded) or whether it is possible to do all of the upgrades and then do a single restart of the Splunkd service at the end?   Thanks, ... View more

I am running a single instance Splunk Enterprise deployment (v. 8.1.3). On the main GUI dashboard, I am getting a Red Health Status of Splunkd flag. On closer inspection, further detail is showing as Index Processor>Buckets with root cause "The percentage of small buckets (71%) created over the last hour is high and exceeded the red thresholds (50%) for index=os, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=11, small buckets=0" What i can't quite figure out is, it is calling this a small bucket alert and yet the number of small buckets created=0. I came across the following search online to do some further checking on this: index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"  | eval bucketSizeMB = round(size / 1024 / 1024, 2)  | table _time splunk_server idx bid bucketSizeMB  | rename idx as index  | join type=left index      [ | rest /services/data/indexes count=0        | rename title as index        | eval maxDataSize = case (maxDataSize == "auto",             750,                                   maxDataSize == "auto_high_volume", 10000,                                   true(),                            maxDataSize)        | table  index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]  | eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))  | eval isSmallBucket     = if (bucketSizePercent < 10, 1, 0)  | stats sum(isSmallBucket) as num_small_buckets          count              as num_total_buckets          by index splunk_server  | eval  percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))  | sort  - percentSmallBuckets  | eval isViolation = if (percentSmallBuckets > 30, "Yes", "No") A Search over the last 24 hours is showing 4 buckets created (and no small buckets) A search over the last 7 days is showing: index="os", total buckets=10, number of small buckets=1 index="_internal", total buckets=38, number of small buckets=1 I guess i am a little intrigued as to why I am seeing this alert as i have had 2 small buckets created in the last week (and the percentage small buckets per index is at worst 10%). Are there any other health checks that i should be looking at on my Indexer? ... View more

I am operating in an environment with a standalone  Splunk Enterprise instance running v8.1.3 on RHEL. In my environment I have around 350 Universal Forwarders that have been up and running for some time. I am running SSL on port 9997 between my forwarders and my Indexer. Certs being used are custom.   I recently have had a problem with two Universal Forwarders. They are not forwarding any information into Splunk.   In the Splunk GUI, they are appearing in Forwarder Management (and if I delete their entries, they reappear again), which looks good. I have two deployment apps pushed down to these forwarders as follows: App1 – indexer_config: Sets outputs.conf to point to indexer and defines clientCert and sslRootCAPath cert. App2 – Splunk_TA_Windows: This App configures inputs.conf to monitor some basic win event logs (e.g System, Security, Application).   Both of the troublesome forwarders are on machines in a dmz and were installed by the same person.   I have looked through the logs on one of the forwarders (see attached PDF). From the logs, it would appear: The connection from the Universal Forwarder to the Deployment Server is working well – I can see it phoning home in the logs and I could also see it downloading the two apps mentioned above. The connection from the Universal Forwarder to the Indexer seems to be having issues – it appears to connect with the indexer but then the indexer forcibly closes the connection for some reason.   I can see error message: “TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<indexer_ip> inside output group default-autolb-group from host_src=<UF_server_hostname> has been blocked” which appears to be relevant."   What I am trying to figure out is whether this is an issue with: The config on the Universal forwarder (possibly an issue with the SSL connection being rejected by the indexer?) An Issue back at the Indexer. ... View more