Hello Experts,
Recently our client decided to ingest data from their database servers to the existing Splunk environment. The existing Splunk environment is like this:
We have already installed Splunk DB Connect 3 in one of the heavy forwarders and we can clearly see the query result from the target database using SQL Explorer. We have also made sure that the HTTP Event Collector port is not blocked or anything so no bind port error in splunkd.log. Several data inputs have also been created. Unfortunately, we still don't know how to forward the events to the indexers.
Here is the outputs.conf we created in $SPLUNK_HOME/etc/system/local:
[default]
defaultGroup = hf_load_balance
[tcpout:hf_load_balance]
compressed = true
server = <idx1>:9997, <idx2>:9997, <idx3>:9997, <idx4>:9997
sslCertPath = /apps/splunk/etc/auth/servercert.pem
sslPassword = $1$bqkDNmCaJfWrxZxBi5bW
sslRootCAPath = /apps/splunk/etc/auth/CoreCA.pem
sslVerifyServerCert = true
And here is one of the inputs created (configuration from db_inputs.conf):
[UXP_Track_Logs]
connection = UXP
description = UXP DB Track_Logs Table
disabled = 0
index = app_uxpdb
index_time_mode = dbColumn
input_timestamp_column_number = 2
interval = 0,15,30,45 * * * *
mode = rising
query = <query for UXP>
query_timeout = 300
sourcetype = uxptracklogs
tail_rising_column_number = 2
The index app_uxpdb exists in all target indexers.
From DB Connect 3 documentation, only indexes.conf configuration is mentioned but not the way to forward the data. Can somebody please guide me for this one?
Thank you.
No, it will not index locally because you already have outputs.conf in place which will send data to all indexers but still blank index is require on heavy forwarder so please create app_uxpdb
index on heavy forwarder and it will send data to all indexers because app_uxpdb
is already present on all indexers based on your question.
No, it will not index locally because you already have outputs.conf in place which will send data to all indexers but still blank index is require on heavy forwarder so please create app_uxpdb
index on heavy forwarder and it will send data to all indexers because app_uxpdb
is already present on all indexers based on your question.
I see. It looks like I missed that one. Thank you.
Please convert your comment to answer so I can accept yours.
Hello vincenteous,
You can do this by sending data over default port 9997 with general configuration and without HEC .
Make sure connection between indexer and UF is proper and also you have created app_uxpdb on both UF and indexers.
Hi,
Index app_uxpdb has already been created in all indexers. It looks like I missed creating a blank index in the HF to trigger the forwarding mechanism as harsmarvania57 pointed out. Thank you for answering.
Your mention of HTTP event collector is throwing me off... 🙂 Port 9997 is the default port for the Splunk-2-Splunk TCP listener and your regular steps to enable forwarding apply.
Can you please clarify your use of the HTTP event collector in this context? I suspect it's just a terminology issue and your problem is likely that you did not enable the SplunkForwarder app.
Hi,
I was following the troubleshooting documentation about HEC in db connect. Here it is: http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshooting#Debug_HTTP_Event_Collector...
At first, because my splunkd port was set to 8088, HEC created by DB Connect cannot bind the default 8088 TCP port. In that case, I changed the port to something else and no error again.
I have checked the splunkd.log and found the entry "Connected to ...." so the connection seems to be fine.
Hi,
Have you created app_uxpdb
index on heavy forwarder on which DB connect is installed?
Hi,
No, I haven't. Won't it cause Splunk to index locally instead? I don't know, maybe I'm mistaken here.