Hello Experts, Currently, I have set up both the search head cluster and the indexer cluster in my production environment and all indexing and searching activities work perfectly. However, there are also standalone indexers outside of my environment which are handled by another team in a different network segment and are required to be added as search peers, but can't be added as indexer cluster members due to administration issues. With this situation, can I just simply add those standalone indexers as search peers for my search head cluster from the GUI? Or is there another additional procedure to be added? Thank you and please advise. ... View more

Hello Experts, Recently our client decided to ingest data from their database servers to the existing Splunk environment. The existing Splunk environment is like this: 3 Search Heads under a load balancer (no cluster) 4 Indexers (no cluster) 3 Heavy Forwarders We have already installed Splunk DB Connect 3 in one of the heavy forwarders and we can clearly see the query result from the target database using SQL Explorer. We have also made sure that the HTTP Event Collector port is not blocked or anything so no bind port error in splunkd.log. Several data inputs have also been created. Unfortunately, we still don't know how to forward the events to the indexers. Here is the outputs.conf we created in $SPLUNK_HOME/etc/system/local: [default] defaultGroup = hf_load_balance [tcpout:hf_load_balance] compressed = true server = <idx1>:9997, <idx2>:9997, <idx3>:9997, <idx4>:9997 sslCertPath = /apps/splunk/etc/auth/servercert.pem sslPassword = $1$bqkDNmCaJfWrxZxBi5bW sslRootCAPath = /apps/splunk/etc/auth/CoreCA.pem sslVerifyServerCert = true And here is one of the inputs created (configuration from db_inputs.conf): [UXP_Track_Logs] connection = UXP description = UXP DB Track_Logs Table disabled = 0 index = app_uxpdb index_time_mode = dbColumn input_timestamp_column_number = 2 interval = 0,15,30,45 * * * * mode = rising query = <query for UXP> query_timeout = 300 sourcetype = uxptracklogs tail_rising_column_number = 2 The index app_uxpdb exists in all target indexers. From DB Connect 3 documentation, only indexes.conf configuration is mentioned but not the way to forward the data. Can somebody please guide me for this one? Thank you. ... View more

Hello Guys, I have a bit of a curious case and it is really bugging our production environment. I have deployed around 12 Windows UF to monitor Security event logs within AD servers which are located behind a firewall. The version of the UFs is 5.0.2 currently and I have set the input and output configurations using a deployment server. From the first deployment, I could see all 12 servers are sending the logs just fine. After several hours, the number of servers dropped to 7. The drop sequence continue until no server is sending logs at all. I tried to use just a single server as a test project and I found that the server is only sending logs for about 3 - 4 hours max before stopped sending completely. No errors or warnings found within splunkd.log of the forwarder and my indexer. The splunkd.log's entries were only "Connected to ...." and "... phone home ....". I also did not see any blocking event from metrics.log My configurations are like this: inputs.conf [WinEventLog://Security] disabled = 0 index = app_ad sourcetype = tseladscrt start_from = oldest current_only = 0 _TCP_ROUTING = loadheavyfwd outputs.conf [tcpout:loadheavyfwd] compressed = true server = <indexerip>:9997 sslCertPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\cert.pem sslPassword = xxxxxxxxxxxxx sslRootCAPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\CoreCA.pem sslVerifyServerCert = true Where should I start to troubleshoot? Thank you. ... View more

Hi all, Recently, I've been tasked to upgrade 2 different Splunk systems (1 server each), version 6.1.7 and version 6.3.8, to version 6.5.0. Both existing systems run on Windows Server 2012 with no internet connection and other running 3rd party software. I performed the upgrade without using silent install method. I could perform an upgrade to 6.5.0 from version 6.1.7 just fine, but it's a different story for version 6.3.8. There were 2 error messages which were shown by the installer during the upgrade activity: The first one was "missing SSLEAY32.DLL" for OpenSSL. Before upgrading, I'd made sure that the existing system already had 2 DLL files in OpenSSL folder (under Python folder) and the registry had already pointed to that particular path. I bypassed this error by quickly copied the DLLs from the previous system to the new system as soon as the folder was created by Splunk. The second error was "missing MSVCR110.DLL". I resolved this one by simply installing Visual C++ 2012 Redistributable. After resolving those 2 errors, the upgrade process continued and was able to finish completely. But, there were error entries in splunkd.log which mentioned checksum error for 2 DLL files which I had copied from the previous system. Because of this, I couldn't execute certain commands such as "splunk list monitor". I then made a small lab to test the upgrade process again. Using the lab system, I found out that every Splunk version 6.3.x and above (excluding 6.5.0) experienced the same type of error (missing SSLEAY32.DLL and MSVCR110.DLL). This thing didn't happen when I installed 6.5.0 directly or performed an upgrade from Splunk version below 6.3.0. Is this a bug? Has anyone ever encountered the same thing as I was? Any idea on resolving the checksum error for OpenSSL's DLL files? With this case, I don't think I can continue to upgrade my other customer's system. Thank you. ... View more