Getting Data In

Cannot remove forward-server


I have inherited an old on-prem Splunk 7.0.2 installation that I'm now trying to reconfigure to forward data to our SplunkCloud instance.
I have installed the SplunkCloud app on the search-head that is acting as deploy-server. It is now forwarding its internal logs to the cloud as expected. Now I want to remove the old forward-servers.

When I execute: ./splunk list forward-server
I get this:

Active forwards:
10.yy.167.67:9997 (ssl)
(ssl) Configured but inactive

This looks correct.
Then I execute: ./splunk remove forward-server 10.yy.167.68:9997 and get this message:

In handler 'tcpout-server': Type = outputs, Context = (user: nobody, app:, root: /opt/splunk/etc), Acting as = nobody: Invalid configuration context: Cannot read configuration if user context is set but app context is not

I get the same kind of message if I try to do this in the web gui.

I have tried to find the configuration file that contains this forward-server config but I cannot locate it.
It should be /splunk/etc/system/local/outputs.conf but there is no such file. I have tried grabbing for the IP address in the entire /etc directory structure but got no matches!

Does anyone have any input on how to proceed?


0 Karma



Try to find out tcpout stanza from outputs.conf using btool and find that server.

$PLUNK_HOME/bin/splunk cmd btool outputs list --debug


Good idea!
I executed that command and found one suspect:



Could this be the issue? Is it safe to remove? We are going to retire the entire on-prem indexer setup, we do already have all data in SplunkCloud.

0 Karma


Server 10.yy.167.67 under any tcpout stanza with parameter server ?

It looks like you are using Indexer discovery and due to that 10.yy.167.67 indexer is coming from Cluster Master, in that case that server will be removed from forward-server list when you'll decommission that indexer from Indexer Cluster.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!


Or Learn More in Our Blog >>