Getting Data In

After disabling a Forwarder, it keeps talking to the receiver

Stefan
Explorer

I had several lightweight forwarders set up, with all of them pointing towards a single Cook Fwd.

Due to a mistake on 1 of the machines (over 190 of them) I wanted to disable forwarding, did so by editing outputs.conf again and restarted.

Communications wouldn't stop, so I started searching for other copies of the file, all were empty.

I then disabled the forwarder through CLI, restarted once again, all to no avail.

After searching for the receiver's IP on the forwarder box, I found it embeded in the outputs.conf that resides inside the search APP. Cleaned it, then restarted.

That did solve the issue for me, but the question is: What gives?

1 Solution

Genti
Splunk Employee
Splunk Employee

its hard to tell what went on with your configuration, however it seems that you were not correct when saying:
"I started searching for other copies of the file, all were empty"
Somehow you had multiple outputs.conf files, the one that you deleted and the one that was residing in the search app.

Next time, you might want to run the following command and make life easier for you:

./splunk cmd btool outputs list --debug

This will tell you exactly what output.conf stanzas exist and where they are residing.

View solution in original post

Genti
Splunk Employee
Splunk Employee

its hard to tell what went on with your configuration, however it seems that you were not correct when saying:
"I started searching for other copies of the file, all were empty"
Somehow you had multiple outputs.conf files, the one that you deleted and the one that was residing in the search app.

Next time, you might want to run the following command and make life easier for you:

./splunk cmd btool outputs list --debug

This will tell you exactly what output.conf stanzas exist and where they are residing.

Stefan
Explorer

Great tip, thanks !!

I had tried looking for copies of the file using "find outputs.conf".That returned several files but not the specific one inside Search...

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...

Splunk Cloud Application Management in Terraform

Now On-Demand   We’re diving into how you can bring Infrastructure as Code (IaC) principles to your Splunk ...

What's New in Splunk Enterprise Security (ES) 8.6

Purpose-Built AI Agents for the Agentic SOC  Splunk Enterprise Security 8.6 expands AI in Security with ...