I have inherited an old on-prem Splunk 7.0.2 installation that I'm now trying to reconfigure to forward data to our SplunkCloud instance.
I have installed the SplunkCloud app on the search-head that is acting as deploy-server. It is now forwarding its internal logs to the cloud as expected. Now I want to remove the old forward-servers.
When I execute: ./splunk list forward-server
I get this:
Active forwards:
10.yy.167.67:9997 (ssl)
inputs1.q.splunkcloud.com:9997
(ssl) Configured but inactive
forwards:
10.yy.167.68:9997
inputs10.q.splunkcloud.com:9997
inputs11.q.splunkcloud.com:9997
inputs12.q.splunkcloud.com:9997
inputs13.q.splunkcloud.com:9997
inputs14.q.splunkcloud.com:9997
inputs15.q.splunkcloud.com:9997
inputs2.q.splunkcloud.com:9997
inputs3.q.splunkcloud.com:9997
inputs4.q.splunkcloud.com:9997
inputs5.q.splunkcloud.com:9997
inputs6.q.splunkcloud.com:9997
inputs7.q.splunkcloud.com:9997
inputs8.q.splunkcloud.com:9997
inputs9.q.splunkcloud.com:9997
This looks correct.
Then I execute: ./splunk remove forward-server 10.yy.167.68:9997
and get this message:
In handler 'tcpout-server': Type = outputs, Context = (user: nobody, app:, root: /opt/splunk/etc), Acting as = nobody: Invalid configuration context: Cannot read configuration if user context is set but app context is not
I get the same kind of message if I try to do this in the web gui.
I have tried to find the configuration file that contains this forward-server config but I cannot locate it.
It should be /splunk/etc/system/local/outputs.conf
but there is no such file. I have tried grabbing for the IP address in the entire /etc directory structure but got no matches!
Does anyone have any input on how to proceed?
Regards,
Andreas
Hi,
Try to find out tcpout
stanza from outputs.conf using btool and find that server.
$PLUNK_HOME/bin/splunk cmd btool outputs list --debug
Good idea!
I executed that command and found one suspect:
[tcpout:clustered_indexers]
[indexer_discovery:cluster_master]
Could this be the issue? Is it safe to remove? We are going to retire the entire on-prem indexer setup, we do already have all data in SplunkCloud.
Server 10.yy.167.67
under any tcpout
stanza with parameter server
?
It looks like you are using Indexer discovery and due to that 10.yy.167.67
indexer is coming from Cluster Master, in that case that server will be removed from forward-server list when you'll decommission that indexer from Indexer Cluster.