topic Splunk_Server is showing different server than the one in outputs.conf in Getting Data In

Splunk_Server is showing different server than the one in outputs.conf

jordanking1992 — Thu, 15 Aug 2019 13:22:07 GMT

Hello,

We currently in the process of moving to indexer clustering with 3 new servers. The 3 old servers are standalone. After setting up the 3 new cluster peers and verify that the command ran for this are correct, i decided to send data to them. In my outputs.conf, i have only specified ONE peer from the new indexer cluster. However, when data is sent and indexed, the search head shows the splunk_server field with a value of our old server There are no references to that server name in any of the conf files either.. My machine has never sent data to the old servers so i am not sure whats going....

Re: Splunk_Server is showing different server than the one in outputs.conf

somesoni2 — Thu, 15 Aug 2019 13:55:16 GMT

In what time range you're searching and is the old indexer not receiving any data in that time frame? Is your search head searching to both old and new indexers?

Re: Splunk_Server is showing different server than the one in outputs.conf

jordanking1992 — Thu, 15 Aug 2019 14:00:52 GMT

Timerange was in the last 15 minutes that i sent demo data. The searchheads are tied to the 3 old servers and the master_node that searches the three new. I am sending data to only one of new (cluster) nodes.

Re: Splunk_Server is showing different server than the one in outputs.conf

somesoni2 — Thu, 15 Aug 2019 14:12:53 GMT

Can you login to Splunk web of that indexer and run the search? Just confirm if the data is indeed ingested on that old indexer or not.

Re: Splunk_Server is showing different server than the one in outputs.conf

jordanking1992 — Thu, 15 Aug 2019 15:11:54 GMT

Weird. Yes, it is in fact indexed on that server and NOT the new cluster node specified in my outputs.conf. There is no reference to that server when using btool to check outputs.conf....

Could this be a bug or some type of network routing issue?

Re: Splunk_Server is showing different server than the one in outputs.conf

jordanking1992 — Wed, 30 Sep 2020 01:44:52 GMT

root@ubuntu:/opt/splunk/etc/system/local# splunk btool outputs list --debug
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf defaultGroup = qa_cluster
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)
/opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/default/outputs.conf indexAndForward = false
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunk/etc/system/local/outputs.conf [tcpout:qa_cluster]
/opt/splunk/etc/system/local/outputs.conf forceTimebasedAutoLB = true
/opt/splunk/etc/system/local/outputs.conf server = susplkidxqa001.homeoffice.anfcorp.com:9997

Re: Splunk_Server is showing different server than the one in outputs.conf

woodcock — Thu, 15 Aug 2019 21:19:10 GMT

/opt/splunk*/bin/splunk btool list outputs --debug

You will see that there is a file somewhere (probably /opt/splunk/etc/system/local/outputs.conf) that is pointing to the old indexer.