Splunk forwarder configured with dummy splunk server results in long pause in puppet splunk restart

[davidheward]

I basically have roles which install the forwarder with whom I might wish to do some local testing.
When testing locally the splunk server name (which is just splunk) does not resolve, which is to be expected.

However i've noticed Splunk then hanging for around 300 seconds whilst it retries TCP/curl of the server name. Is there a smart config value I can apply to tell the splunk service to not try and find its splunk server up-front?

Here's what my outputs.conf looks like

[default]
defaultGroup=splunk_9997
disabled=true
dnsResolutionInterval=5

[tcpout:splunk_9997]
server=splunk:9997
disabled=true
dnsResolutionInterval=5

[indexer_discovery:splunk_9997]
cxn_timeout=1

And some previous example logs, NOTE these logs are NOT with the above outputs.conf but rather a conf that included no work arounds for disabled, dnsResolutionInterval and cxn_timeout

06-29-2017 12:06:23.100 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: splunk
06-29-2017 12:11:23.055 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: splunk
06-29-2017 12:13:04.496 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 100 seconds.
06-29-2017 12:14:44.515 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 200 seconds.
06-29-2017 12:16:23.138 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: splunk
06-29-2017 12:16:24.531 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 300 seconds.
06-29-2017 12:18:04.544 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 400 seconds.
06-29-2017 12:19:44.558 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 500 seconds.
06-29-2017 12:21:23.045 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: splunk
06-29-2017 12:21:24.570 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 600 seconds.
06-29-2017 12:23:04.584 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for 700 seconds.
06-29-2017 12:24:44.599 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunk_9997 blocked for

[gcusello]

Hi davidheward,
why you don't use IP address instead hostname to address your indexers?

In addition Splunk Best Practices suggest to not locally configure ouputs.conf but instead at the installation, configure only Deployment Server and deploy outputs.conf in a dedicated TA using Deployment Server.
In this way you easier configure any variation of you indexers addresses (add oremove indexers, ...).

Bye.
Giuseppe

[davidheward]

Hi Cusello,

Thanks for the reply.
How will adding an IP help not "attempting" to communicate with the indexers?

Interesting second point. Can you point me to some documentation that explains a little more about what you mean by "TA using deployment server".

I'm putting the outputs.conf down with puppet atm.

[gcusello]

Hi davidheward,
I don't understand why you want to configure an indexers not communicating: if you want to test connection, you have to configure a correct Indexer otherwise you don't need to configure an indexer in outputs.conf.
The best way is to create a TA that contains only outputs.conf and apps.conf and deploy it using a Deployment server: it's the more efficient way to manage forwarders.

I haven't documentation: my teachers explainded this in a course.

TA's structure could be this:

• bin (empty folder)
• default (contains outputs.conf and apps.conf)
• metadata (empty folder)
• static (empty folder)

bye.
Giuseppe