Hello Experts, Currently, I have set up both the search head cluster and the indexer cluster in my production environment and all indexing and searching activities work perfectly. However, there are also standalone indexers outside of my environment which are handled by another team in a different network segment and are required to be added as search peers, but can't be added as indexer cluster members due to administration issues. With this situation, can I just simply add those standalone indexers as search peers for my search head cluster from the GUI? Or is there another additional procedure to be added? Thank you and please advise. ... View more

Hello Experts, Recently our client decided to ingest data from their database servers to the existing Splunk environment. The existing Splunk environment is like this: 3 Search Heads under a load balancer (no cluster) 4 Indexers (no cluster) 3 Heavy Forwarders We have already installed Splunk DB Connect 3 in one of the heavy forwarders and we can clearly see the query result from the target database using SQL Explorer. We have also made sure that the HTTP Event Collector port is not blocked or anything so no bind port error in splunkd.log. Several data inputs have also been created. Unfortunately, we still don't know how to forward the events to the indexers. Here is the outputs.conf we created in $SPLUNK_HOME/etc/system/local: [default] defaultGroup = hf_load_balance [tcpout:hf_load_balance] compressed = true server = <idx1>:9997, <idx2>:9997, <idx3>:9997, <idx4>:9997 sslCertPath = /apps/splunk/etc/auth/servercert.pem sslPassword = $1$bqkDNmCaJfWrxZxBi5bW sslRootCAPath = /apps/splunk/etc/auth/CoreCA.pem sslVerifyServerCert = true And here is one of the inputs created (configuration from db_inputs.conf): [UXP_Track_Logs] connection = UXP description = UXP DB Track_Logs Table disabled = 0 index = app_uxpdb index_time_mode = dbColumn input_timestamp_column_number = 2 interval = 0,15,30,45 * * * * mode = rising query = <query for UXP> query_timeout = 300 sourcetype = uxptracklogs tail_rising_column_number = 2 The index app_uxpdb exists in all target indexers. From DB Connect 3 documentation, only indexes.conf configuration is mentioned but not the way to forward the data. Can somebody please guide me for this one? Thank you. ... View more

Hello Guys, I have a bit of a curious case and it is really bugging our production environment. I have deployed around 12 Windows UF to monitor Security event logs within AD servers which are located behind a firewall. The version of the UFs is 5.0.2 currently and I have set the input and output configurations using a deployment server. From the first deployment, I could see all 12 servers are sending the logs just fine. After several hours, the number of servers dropped to 7. The drop sequence continue until no server is sending logs at all. I tried to use just a single server as a test project and I found that the server is only sending logs for about 3 - 4 hours max before stopped sending completely. No errors or warnings found within splunkd.log of the forwarder and my indexer. The splunkd.log's entries were only "Connected to ...." and "... phone home ....". I also did not see any blocking event from metrics.log My configurations are like this: inputs.conf [WinEventLog://Security] disabled = 0 index = app_ad sourcetype = tseladscrt start_from = oldest current_only = 0 _TCP_ROUTING = loadheavyfwd outputs.conf [tcpout:loadheavyfwd] compressed = true server = <indexerip>:9997 sslCertPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\cert.pem sslPassword = xxxxxxxxxxxxx sslRootCAPath = D:\Program Files\SplunkUniversalForwarder\etc\auth\CoreCA.pem sslVerifyServerCert = true Where should I start to troubleshoot? Thank you. ... View more