Hi, We have a Splunk app which exposes a REST end point for other application to request metrics. The main piece of python code inside the method is: service = self.getService() searchjob = service.jobs.create(searchquery) while not searchjob.is_done(): time.sleep(5) reader = results.ResultsReader(searchjob.results(count=0)) response_data = {} response_data["results"] = [] for result in reader: if isinstance(result, dict): response_data["results"].append(result) elif isinstance(result, results.Message): mylogger.info("action=runSearch, search = %s, msg = %s" % (searchquery, results.Message)) search_dict["searchjob"] = searchjob search_dict["searchresults"] = json.dumps(response_data) The dependent application invokes the REST API at some scheduled intervals. There are close to 150 calls that is spread across various time intervals. Note: At any point of time there will be maximum 6 search requests. Normal scenarios Remote application and my Splunk app - both are up and running - everything is fine. For some reason if I have to restart remote application, and after restart - both are up and running - everything is fine. For some reason if I have to restart my Splunk process, and after restart - both the applications are up and running - everything is fine. Problematic scenario: The problem starts when the system where remote application is running is rebooted. After reboot, the remote application will start making calls to the splunk application and in about 60 min, the number of CLOSE_WAIT connections reaches 700+ and eventually splunk system starts throwing socket error. Splunk Web will also become inaccessible. Additional Info: The remote application is a python application written using Tornado framework. The remote application runs inside a docker container that is managed by Kubernetes. The ulimit -n on splunk system shows 1024. (I know that as per Splunk recommendation it is less. But i would like to understand why the issue occurs only during remote systemreboot) During normal times, the searches take on an average 7s to complete. When the remote machine is rebooted during that time the searches take on an average 14s to complete. (Well this may not make sense to relate remote system reboot with splunk search performance on the splunk system. But thats the trend) The CLOSE_WAIT connections are all internal tcp connections tcp 1 0 127.0.0.1:8089 127.0.0.1:37421 CLOSE_WAIT 0 167495826 28720/splunkd tcp 1 0 127.0.0.1:8089 127.0.0.1:32869 CLOSE_WAIT 0 167449474 28720/splunkd tcp 1 0 127.0.0.1:8089 127.0.0.1:37567 CLOSE_WAIT 0 167497280 28720/splunkd tcp 1 0 127.0.0.1:8089 127.0.0.1:33086 CLOSE_WAIT 0 167451533 28720/splunkd Any help or pointers is highly appreciated. Thanks, Strive ... View more

Hi, Our monitor configuration is: [monitor:///opt/diags.log*] disabled = false host = $decideOnStartup sourcetype = diag_snapshot blacklist = \.(gz)$ index = my_index initCrcLength = 1024 ignoreOlderThan = 12h When we deploy this configuration, there is a log file whose modtime is older than 12 hours. Now when I restart Splunk, it checks the modtime and notices that it is older than 12 hours and it doesn't forward the data to indexer. After few hours some lines are written to the log file. The Splunk forwarder is not sending that log file to indexer. According to Splunk documentation - A file whose modtime falls outside this time window when seen for the first time will not be indexed at all. Is there way to make Splunk forwarder forward the newly written log lines to the indexer? For a fact, I know that if I restart Splunk again, the new lines will be indexed. Please note that it is a production system and manually monitoring files and restarting Splunk is not possible. Please let me know if there is any setting that will work along with ignoreOlderThan. Thanks, Strive ... View more

Hi, We have a requirement to forward logs from clients (Splunk universal Forwarders) to a server using SSL (tls1.2) First Try: We installed same server certificate on both server and clients (as mentioned in the examples in splunk documentation and in splunk blogs). It worked fine. Change request: Each client should have its own client certificate. Second Try: We created multiple client certificates, one for each client. Installed those certificates on the client. We started getting a error: connection is not established. For second try, we followed the below mentioned steps Step 1: Created a CA Certificate - CACert.pem Step 2: Created a Server Certificate using the above CA Certificate - ServerCert.pem Step 3: Created four client certificates using the above CA Certificate - Client1Cert.pem, Client2Cert.pem, Client3Cert.pem, and Client4Cert.pem Step 4: Installed certificates Step 5: Restarted Splunk on server and clients. ...... Started seeing connection related errors in splunkd.log ............ Client1 outputs.conf file [tcpout] disabled = false defaultGroup = mac [tcpout:mac] server = xxx.xxx.xxx.xxx:YYYY sslRootCAPath = $SPLUNK_HOME/etc/certs/CACert.pem sslCertPath = $SPLUNK_HOME/etc/certs/Client1Cert.pem sslPassword = Client1_privkey_password sslVerifyServerCert = true sslCommonNameToCheck=commonnametest Client2 outputs.conf file [tcpout] disabled = false defaultGroup = mac [tcpout:mac] server = xxx.xxx.xxx.xxx:YYYY sslRootCAPath = $SPLUNK_HOME/etc/certs/CACert.pem sslCertPath = $SPLUNK_HOME/etc/certs/Client2Cert.pem sslPassword = Client2_privkey_password sslVerifyServerCert = true sslCommonNameToCheck=commonnametest similarly for other clients...... Server inputs.conf file [SSL] rootCA = $SPLUNK_HOME/etc/certs/CACert.pem serverCert = $SPLUNK_HOME/etc/certs/ServerCert.pem password = server_privkey_password requireClientCert = true sslVersions = tls1.2 allowSslRenegotiation = true [splunktcp-ssl:YYYY] How will universal forwarder clients validate that the server certificate that is presented is valid? Similarly, how will the server validate that the client certificate that is presented is valid? What is wrong here? Could you please help. Thanks, Strive ... View more