How do I fix a large amount of duplicate events that are locking out my instance?

[stuartrking]

I've been tasked with installing Splunk Cloud on our hosted Windows environment, and I'm running into issues getting all of the forwarding working properly.

I have two Universal Forwarders sending data to a Heavy Forwarder acting as a Gateway Forwarder. This Gateway Forwarder is then communicating with our cloud SaaS.

On the Universal Forwarder instance I get the following error:

No connection could be made because the target machine actively refused it.

The problem is I already have the Gateway Forwarder set to accept connections on this port, and additionally, there are no firewall rules to block the communication.

The logs on the Gateway Forwarder report that essentially all of the logs coming through it are possible duplicates, and after some point, the cloud SaaS blocks communications temporarily.

This duplicate entry issue is appearing for Splunk's own logs as well as the logs for our application. I've tried reinstalling the Universal Forwarders, but are there any other steps that I could follow or configurations that I could change?

Thanks in advance!

[lguinn2 · Accepted solution]

Okay, on your universal forwarders (UFs) - somewhere in a "local" directory, you must have an outputs.conf that tells the UFs to send the data to the heavy forwarder. That outputs.conf file should look something like this, at a minimum

outputs.conf

[tcpout:default_group]
server=10.1.1.197:9997

This assumes that your heavy forwarder is at ip adddress 10.1.1.197 and that it is listening for forwarders to send it data on poirt 9997.
Note that port 9997 must not be configured to do anything else - this is for splunk-to-splunk communications. However, you can choose some other port: 9997 is just what I am using in this example.

The heavy forwarder must be listening for inputs from forwarders. So inputs.conf on the heavy forwarder should have the following, at a minimum

inputs.conf

[splunktcp://:9997]

This establishes the communication from your universal forwarders to your heavy forwarder. Since this is your gateway forwarder, it should have an app installed that lets it communicate with your servers in the Splunk Cloud.

On the universal forwarders and on the heavy forwarder, take a look at splunkd.log - this log file will give you great information on what the component is doing and any errors or failures. You can find the log in $SPLUNK_HOME/var/log/splunk

[stuartrking]

It looks like the outputs.conf was my issue. My outputs.conf looked something like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = (Gateway Forwarder IP):9997

[tcpout-server (Gateway Forwarder IP)://:9997]

I had never changed the config manually and the only time I set the Gateway Forwarder IP addresses was in the initial installation. Any ideas how it could have been set to the same value over two separate installations?

Thank you again!

[stuartrking]

Apologies for the confusion. This answer is all set. Further investigation showed that our product's logs were guilty of producing duplicate entries. Thank you for your help!

[lguinn2]

It would be helpful to see how you have configured the inputs.conf on the Gateway Forwarder, as well as the outputs.conf on the Universal Forwarders. Feel free to obfuscate server names and/or ip address...

[stuartrking]

Thank you for following up,

The outputs.conf is using the default properties at the moment. The only configuration that I set up was the IP address of the Indexer and the Deployment server. I set both of those to the IP of the Gateway Forwarder with their respective default ports. Would that cause any problems?

The Gateway Forwarder outputs.txt only has tcpout:splunk cloud and the SSL password set. Otherwise the defaults are being used.

Are there any parameters that you suggest I change from default?

[lguinn2]

Sorry, but the community really needs to see the outputs.conf of the Universal Forwarders. The Deployment server should not appear in the outputs.conf file on any forwarder.

It is fine to use the gateway forwarder as a deployment server. Eventually, you may outgrow that, and then spin up a separate deployment server. But combining both functions is fine. That is not the problem.

I don't think that you need to change any default configurations. I am questioning whether your basic settings are correct.

[stuartrking]

That's the thing, there is no outputs.conf in my local configuration directory. The default values are as follows:

Version 6.4.0

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false

I set the files that I wanted to follow using the CLI, not configuration files. Would an export of the monitored files help as well? That only has a list of the directories.

[lguinn2]

This is not a problem about your monitored files - the question is, what does the forwarder do with the files? Where should it send them? That's why I keep asking about outputs.conf - it defines where the data goes. If it isn't specified, your data goes nowhere. If it is specified incorrectly, your data might go twice.

[lguinn2]

Although I suppose that you could have specified the same inputs twice, in such a way that Splunk collects the data twice - but that's pretty hard to do unless you have a lot of links and complicated stanzas in inputs.conf. That's a different direction for the investigation.