Getting Data In
Highlighted

Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

Hi,

We recently had to deploy a heavy forwarder into the Splunk architecture.
Last time, the flow was from a source->IDX.directly. Now we had to deploy like this : source-> HF-> IDX.
We have deployed and configured the heavy forwarder with forwarder license and configurations as follows:

*--Outputs.conf—

[tcpout]
defaultGroup = SplunkIdx
[tcpout: SplunkIdx]
Server = < SplunkIdxIPAddress>:9997
[tcpout-server://< SplunkIdxIPAddress>:9997>]*

The data input was configured on the heavy forwarder for the receiving from TCP port 514.
As for the indexer, the receiving port has been configured to accept 9997 port.
All of the firewall rules are provisioned accordingly. We tested the connection using netstat and telnet, all is alright.
One more thing to note:
Last time they were using index=emailsource.
Now, on the heavy forwarder, on the data input configuration, we have also used the same settings used by the indexer last time. The index name is also the same (emailsource).

Now, when we point the source to the heavy forwarder, we cannot see any logs/event coming in.
We check the splunkd logs,within the time frame, and we cannot see any error messages regarding anything.

Any possible cause as to why there are not events also coming in?

Thanks

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

Sorry about the error on the post:

--Outputs.conf—
[tcpout]
defaultGroup = SplunkIdx
[tcpout: SplunkIdx]
Server = < SplunkIdxIPAddress>::9997
[tcpout-server://< SplunkIdxIPAddress>:9997>]

I would like to add that the heavy forwarder will not be indexing any data.

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

SplunkTrust
SplunkTrust

Do you see the HF's internal logs in your indexer (search index=_internal source=*splunkd.log host=<HF name or address>)? This will confirm if the connection from HF to indexer is functioning.
Please share the inputs.conf settings on the HF.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

@richgalloway I searched for events using the one you provided. the indexer/search head gives out some outputs like the one below:

WARN TcpInputConfig - reverse dns lookups appear to be excessively slow, this may impact receiving from network inputs. 10.524853 % time is greater than configured rdnsMaxDutyCycle=10 %. Currently lookup: host::10.xx.xx.xx

For the inputs.conf,

[default]
host = <hostname of HF>

[tcp://514]
index = emailsource
disabled = 0
0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

SplunkTrust
SplunkTrust

The cited log message doesn't tell us if events are being received from the HF.

I suggest you resolve the DNS problem first, however.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

@richgalloway I will go ahead and have this error cleaned up first also. Thank you so much.

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Ultra Champion

In your outputs.conf it looks like you have a space in "tcpout: SplunkIdx" - you don't want that!

try this:

[tcpout]
defaultGroup = SplunkIdx

[tcpout:SplunkIdx]
server = SplunkIdxIPAddress:9997

you also dont need [tcpout-server://< SplunkIdxIPAddress>:9997>]

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

@nickhillscpl apologies for the typo but i double-checked and there is no space between the "tcpout:" and "SplunkIdx". I will take note of this syntax in the future, however. thanks for the warning.

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

SplunkTrust
SplunkTrust

Are you receiving Splunk-internal events though? What does splunkd.log say?

0 Karma
Highlighted

Re: Why are new events not coming in when a new heavy forwarder is deployed to the architecture?

Explorer

@skalliger I am only receiving the error like the one below:

WARN TcpInputConfig - reverse dns lookups appear to be excessively slow, this may impact receiving from network inputs. 10.524853 % time is greater than configured rdnsMaxDutyCycle=10 %. Currently lookup: host::10.xx.xx.xx

So far, there are no other warning or error messages indicated on the splunkd.log.
I will try to work on the dns resolution also. See if that is the reason.
Would there be a chance that there are some other things to consider, aside from the outputs.conf and input.conf? By the way, the requirement is just to pump out all the logs coming into the HF, down to the IDX, with no consideration as to which logs are brought in and passed out.

0 Karma