Deployment client's phoneHomeIntervalInSecs setting is not being honored according to splunkd_access.log

[Ellen]

In my deploymentclient.conf I have added the phoneHomeIntervalInSecs to be 1800 seconds (30 minutes) to override the default of 60 seconds.

eg. $SPLUNK_HOME/etc/apps/TestApp/local/deploymentclient.conf

[deployment-client]
phoneHomeIntervalInSecs = 1800

[target-broker:deploymentServer]
targetUri= mydeploymentserver.acme.com:8089

The deployment server is working since I see my various apps with my updated configuration files on my client host but according to the following search:

index=_internal source=*splunkd_access.log POST phonehome

the phonehome connection entries show up every 60 seconds as oppose to every 30 minutes in

$SPLUNK_HOME/var/log/splunk/splunkd_access.log

It looks like the phoneHomeIntervalInSecs settings is still set at 60 seconds. I restarted the deployment client and deployment server Splunk instances. Why is that?

Is it because I moved
$SPLUNK_HOME/etc/system/local/deploymentclient.conf into my app location: $SPLUNK_HOME/etc/apps/TestApp/local

Does the configuration file have to reside in the $SPLUNK_HOME/etc/system/local directory?

[Ellen · Accepted solution]

The precedence of configuration files would take place. So in your case, placing the deploymentclient.conf in your TestApp/local from system/local is fine.

What you have encountered is a known issue (SPL-41174) where
the phoneHomeIntervalInSecs setting is actually being honored despite splunkd_access.log's "POST /services/broker/phonehome/connection_....." message displaying every 60 seconds. This was seen under 4.2 +

Workaround:

The deploymentclient.conf should have the phoneHomeIntervalInSecs setting in both stanzas: [deployment-client] and [target-broker:deploymentServer]
Splunk will need to be restarted on the client host.

eg.
[deployment-client]
phoneHomeIntervalInSecs = 1800

[target-broker:deploymentServer]
targetUri= mydeploymentserver.acme.com:8089
phoneHomeIntervalInSecs = 1800

Additionally you can always confirm the deployment client's phoneHomeIntervalInSecs is being properly set by temporarily enabling DEBUG logging on the deployment server and reviewing splunkd.log for "phoneHome" entries.

On the deployment server via SplunkWeb, can navigate to Manager > System Settings > System Logging > in the query box enter in: deployment *

For each of the 2 entries below, drill down and reset the logging level from the default WARN to DEBUG and Save

1) DeploymentServer

2) DeploymentServerAdminHandler

The logging change will occur dynamically. So there is no need to restart Splunk.

Note: if you restart Splunk, these log levels will revert back to their default.

Let's reload what is memory on the deployment server via:

$SPLUNK_HOME/bin/splunk reload deploy-server

Now on the deployment server via Search, query for the last 60 minutes

index=_internal source=*splunkd.log phonehome

This should display all the DEBUG entries in splunkd.log relating to phonehome and you should see 3 entries from the deployment client along the lines of:

DEBUG DeploymentServer - Recvd phonehome from: ip=.....
DEBUG DeploymentServer - Token matched:....
DEBUG DeploymentServer - Replying back to client[ip=.....

Check the timelines of a set of these to see if they repeat every X seconds based on your specified phoneHomeIntervalInSecs setting.