If you are using unit files for splunk service, then check the value that is set for Restart option. If the value is on-failure, then it wont be restarted upon clean exit. The service will be restarted when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation (such as service reload) times out, and when the configured watchdog timeout is triggered. Trying setting it to always and see what happens. If you are going to use always then ensure that ExecStop=/opt/splunk/bin/splunk stop is also part of your unit file. ... View more

Hi Keerthana, If your fixed sourcetype license is wildcard based then you should have Splunk Version >= 6.3. It would be nice starting point to check your Splunk version and then upgrade it if necessary. When the correct licenses are not available the indexers will be added under the pool whichever seems to fit the bill 🙂 In your case it was Reset License as it would not have restrictions. Check Splunk Version and then proceed further. Regards, Strive ... View more

The issue is resolved. It was due to the _GLOBAL_DEFAULT_TIMEOUT of socket.py To fix, we are setting timeout explicitly before opening the the connection ... View more

The issue would be due to Global Timeout parameters set in some python libraries. Its better to set timeout explicitly. Refer: https://answers.splunk.com/answers/477671/how-to-resolve-lots-of-splunk-internal-connections.html ... View more

The issue that you're seeing is because of you're using ignoreOlderThan setting. When Splunk restarts, it builds as monitoring list (list of files/directories it'll monitor) and a ignore list (list of files and directories which are inside monitored directory OR following monitoring stanza, but should not be monitored due to filter settings like blacklist/ignoreOlderThan etc). When the Splunk started, your file modification time was older than ignoreOlderThan setting hence it made the ignore list. The files in ingore list will not be checked again till the forwarder starts again OR monitoring configuration as reloaded. So option for you would be either to increase the ignoreOlderThan settings (generally it should be same OR more that the frequency in which file gets updated. In your case the file modification frequency over 12h sometimes so keeping it fixed 12h will not work always), OR setup a scripted input that will either restart OR reload the data input configuration. You can reload data input configuration using CLI $SPLUNK_HOME/bin/splunk _internal call /services/data/inputs/monitor/_reload -auth admin:YourAdminPwdOnFwd ... View more

requireClientCert = false , makes it work. As per this link , setting "requireClientCert = true" would require the following conditions to be met : a) "rootCA" must point to a file containing the CA's public key. b) The forwarder's server certificate defined by "sslCertPath" in outputs.conf is signed by that CA. c) The forwarder has the password to read its own certificate ("sslPassword" in outputs.conf). In our case, we were meeting all the conditions but still we faced issues. In the same link, there is point which says -- "The purpose of this requireClientCert=true is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer." So, here is my observation: requireClientCert = true should be set, when we are using same signed certificate on the server(receiver) and on all the clients (forwarders) ... View more

I ran into this and thought I'd post my solution. Just change the rc script per this: http://lists.freebsd.org/pipermail/freebsd-current/2012-January/031246.html. Just change rcvar to "splunk_enable". root@fbsd3:~ # diff -u /etc/rc.d/splunk.orig /etc/rc.d/splunk --- /etc/rc.d/splunk.orig 2014-11-07 17:21:12.000000000 -0600 +++ /etc/rc.d/splunk 2014-11-08 09:24:36.000000000 -0600 @@ -12,7 +12,7 @@ name=splunk extra_commands="status" -rcvar=`set_rcvar` +rcvar=splunk_enable eval "${rcvar}=${${rcvar}:-'NO'}" splunk_start() root@fbsd3:~ # ... View more