After upgrading my DS to Enterprise 9.2.2 clients can't connect

[dionrivera]

After upgrading my deployment server to Enterprise 9.2.2 the clients are no longer connecting to the deployment server. When I launch my DS UI and check for clients connecting, it says 0. Has anyone had this issue?

[dionrivera]

My issue was with the 3 new internal indexes that Splunk Enterprise introduces. In short, my fix was to add the line  selectiveIndexing = true in the %SplunkHome%/etc/system/local/outputs.conf file. Here is a link in the docs referring to this fix. Otherwise, I'm including the synopsis of the symptom/fix from the link I provided initially. Hope that helps.

Resolution

What causes symptom 1?

Splunk Enterprise 9.2.0 introduces a scalable Deployment Server (DS) feature, which makes the DS tier more resilient and highly available. Under the hood, several new internal indexes are introduced to accommodate this feature:

_dsphonehome 
_dsclient
_dsappevent

These indexes are defined in Splunk Enterprise 9.2.x by default. If your DS forwards its data to remote indexers and the indexers are running an older Splunk version, the latter will not have the above-mentioned indexes defined. This will result in the DS being unable to forward and search its DS/DC-related events. The DS's Forwarder Management UI is then unable to list the Deployment Clients (DCs), despite the clients phoning home without any issue.

Fix for symptom 1:

The idea behind it is simple: As long as your DS can index its DS/DC events to the 3 indexes above and search them back, your clients should appear in the UI.

Steps:

1. Allow your DS to selectively index the phone home, client and app events to itself. This is especially applicable to on-prem DS that forwards data to Splunkcloud indexers, but it can be applied to a completely on-prem/cloud BYO environment as well. 

Add the following parameters and values to the DS's outputs.conf file, followed by restarting the splunkd service.

[indexAndForward]
index = true
selectiveIndexing = true

2. This step is applicable if your DS is forwarding its data to on-prem indexing tier and the indexers' version is older than 9.2.0:

Configure the 3 indexes mentioned earlier on your indexing tier. If your indexers are non-clustered, add the index definitions on each of them manually or using your preferred automation. If your indexers are clustered, push the index definitions from the Cluster Manager and enable replication (repFactor = auto) to benefit from cluster redundancy.

Sample indexes.conf configuration:

[_dsphonehome]
homePath   = $SPLUNK_DB/_dsphonehome/db
coldPath   = $SPLUNK_DB/_dsphonehome/colddb
thawedPath = $SPLUNK_DB/_dsphonehome/thaweddb

# clustered indexers only #
repFactor = auto

[_dsappevent]
homePath   = $SPLUNK_DB/_dsappevent/db
coldPath   = $SPLUNK_DB/_dsappevent/colddb
thawedPath = $SPLUNK_DB/_dsappevent/thaweddb

# clustered indexers only #
repFactor = auto

[_dsclient]
homePath   = $SPLUNK_DB/_dsclient/db
coldPath   = $SPLUNK_DB/_dsclient/colddb
thawedPath = $SPLUNK_DB/_dsclient/thaweddb

# clustered indexers only #
repFactor = auto

There is one additional step only if your DC sends its data to the indexers via an intermediate forwarder AND your intermediate forwarder's version is older than 9.2.x:

Add the following parameter and value to the intermediate forwarder's outputs.conf file, followed by a splunkd service restart.

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

At this point, the deployment clients should appear in the Forwarder Management UI > Clients tab.

Tips:

• If you still can't see the clients, run the following query on the DS and check whether it returns some events: 

index=_ds*

• If the query returns nothing and your DS is also a Distributed Monitoring Console instance, go to Settings >  Monitoring Console > Settings > General Setup. Locate your DMC (This instance) and click Edit > Edit Server Roles. Tick the Indexer role and click Save. Run the query again to confirm it is working.

[dionrivera · Accepted solution]

I found the issue described in Symptom 1 of this link https://splunk.my.site.com/customer/s/article/No-Clients-Showing-up-on-Deployment-Server-After-Upgra...

Resolved!

[jaredrissler]

The link seems to be broken. What was the solution? Thanks.

[dionrivera]

My issue was with the 3 new internal indexes that Splunk Enterprise introduces. In short, my fix was to add the line  selectiveIndexing = true in the %SplunkHome%/etc/system/local/outputs.conf file. Here is a link in the docs referring to this fix. Otherwise, I'm including the synopsis of the symptom/fix from the link I provided initially. Hope that helps.

Resolution

What causes symptom 1?

Splunk Enterprise 9.2.0 introduces a scalable Deployment Server (DS) feature, which makes the DS tier more resilient and highly available. Under the hood, several new internal indexes are introduced to accommodate this feature:

_dsphonehome 
_dsclient
_dsappevent

These indexes are defined in Splunk Enterprise 9.2.x by default. If your DS forwards its data to remote indexers and the indexers are running an older Splunk version, the latter will not have the above-mentioned indexes defined. This will result in the DS being unable to forward and search its DS/DC-related events. The DS's Forwarder Management UI is then unable to list the Deployment Clients (DCs), despite the clients phoning home without any issue.

Fix for symptom 1:

The idea behind it is simple: As long as your DS can index its DS/DC events to the 3 indexes above and search them back, your clients should appear in the UI.

Steps:

1. Allow your DS to selectively index the phone home, client and app events to itself. This is especially applicable to on-prem DS that forwards data to Splunkcloud indexers, but it can be applied to a completely on-prem/cloud BYO environment as well. 

Add the following parameters and values to the DS's outputs.conf file, followed by restarting the splunkd service.

[indexAndForward]
index = true
selectiveIndexing = true

2. This step is applicable if your DS is forwarding its data to on-prem indexing tier and the indexers' version is older than 9.2.0:

Configure the 3 indexes mentioned earlier on your indexing tier. If your indexers are non-clustered, add the index definitions on each of them manually or using your preferred automation. If your indexers are clustered, push the index definitions from the Cluster Manager and enable replication (repFactor = auto) to benefit from cluster redundancy.

Sample indexes.conf configuration:

[_dsphonehome]
homePath   = $SPLUNK_DB/_dsphonehome/db
coldPath   = $SPLUNK_DB/_dsphonehome/colddb
thawedPath = $SPLUNK_DB/_dsphonehome/thaweddb

# clustered indexers only #
repFactor = auto

[_dsappevent]
homePath   = $SPLUNK_DB/_dsappevent/db
coldPath   = $SPLUNK_DB/_dsappevent/colddb
thawedPath = $SPLUNK_DB/_dsappevent/thaweddb

# clustered indexers only #
repFactor = auto

[_dsclient]
homePath   = $SPLUNK_DB/_dsclient/db
coldPath   = $SPLUNK_DB/_dsclient/colddb
thawedPath = $SPLUNK_DB/_dsclient/thaweddb

# clustered indexers only #
repFactor = auto

There is one additional step only if your DC sends its data to the indexers via an intermediate forwarder AND your intermediate forwarder's version is older than 9.2.x:

Add the following parameter and value to the intermediate forwarder's outputs.conf file, followed by a splunkd service restart.

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

At this point, the deployment clients should appear in the Forwarder Management UI > Clients tab.

Tips:

• If you still can't see the clients, run the following query on the DS and check whether it returns some events: 

index=_ds*

• If the query returns nothing and your DS is also a Distributed Monitoring Console instance, go to Settings >  Monitoring Console > Settings > General Setup. Locate your DMC (This instance) and click Edit > Edit Server Roles. Tick the Indexer role and click Save. Run the query again to confirm it is working.