I have inherited an old on-prem Splunk 7.0.2 installation that I'm now trying to reconfigure to forward data to our SplunkCloud instance.
I have installed the SplunkCloud app on the search-head that is acting as deploy-server. It is now forwarding its internal logs to the cloud as expected. Now I want to remove the old forward-servers.
When I execute:
./splunk list forward-server
I get this:
(ssl) Configured but inactive
This looks correct.
Then I execute:
./splunk remove forward-server 10.yy.167.68:9997 and get this message:
In handler 'tcpout-server': Type = outputs, Context = (user: nobody, app:, root: /opt/splunk/etc), Acting as = nobody: Invalid configuration context: Cannot read configuration if user context is set but app context is not
I get the same kind of message if I try to do this in the web gui.
I have tried to find the configuration file that contains this forward-server config but I cannot locate it.
It should be
/splunk/etc/system/local/outputs.conf but there is no such file. I have tried grabbing for the IP address in the entire /etc directory structure but got no matches!
Does anyone have any input on how to proceed?
Try to find out
tcpout stanza from outputs.conf using btool and find that server.
$PLUNK_HOME/bin/splunk cmd btool outputs list --debug
I executed that command and found one suspect:
Could this be the issue? Is it safe to remove? We are going to retire the entire on-prem indexer setup, we do already have all data in SplunkCloud.
10.yy.167.67 under any
tcpout stanza with parameter
It looks like you are using Indexer discovery and due to that
10.yy.167.67 indexer is coming from Cluster Master, in that case that server will be removed from forward-server list when you'll decommission that indexer from Indexer Cluster.