Splunk UF clients phoned but are not indexing logs

[mudragada]

I've deployed the deployment-app on the deployment client from deployment server.

The server appeared on the phoned list. But is not indexing the logs.

The splunkforwarder logs don't show any error and the splunk user has the permissions to read the logs.
Where can I check what is not causing the indexing?

[skoelpin]

Did you specify and inputs.conf and outputs.conf in your deployment-apps?

Login to one of your forwarders where this was deployed on. $SPLUNK_HOME/etc/apps/<YOUR_APP>/local

The inputs.conf file should be pointing to a log file which is sitting on the machine. It should also specify the index and sourcetype. Next inspect the outputs.conf file and verify its pointing to an indexer(s). It looks like you've already determined that there is an open port between forwarder and indexer aswell.

Lastly, you need to restart the forwarders for the new conf changes to take affect

[mudragada]

Yes, checked that. it has the inputs.conf as well and outputs.conf is pointing to the indexer.
And the ports are open - there isn't a read error or connection refused in the splunkd logs.

[skoelpin]

Did you restart the Splunk Forwarder service after making these changes?

$SPLUNK_HOME/bin/splunk restart

[mudragada]

Did the restart. Still didn't. At this point I'm looking at, how I can debug the scenario?

[masonmorales]

Does the data show up if you search "all time" against the index? If it does, you may have future-dated data, which would be a parsing issue. (i.e. TZ in props.conf)

[woodcock · Accepted solution]

Did you deploy an app that contains an inputs.conf? Did you configure the forwarder to point to an indexer (or deploy an app with outputs.conf)? Did you set the restart splunkd bit on the app that contains the inputs.conf file (or restart splunk on the forwarder)?

[mudragada]

Yes, that's done. I find the app in the /opt/splunkforwarder with the right hostname, index name and all in the inputs.conf. I'm still not seeing the logs. I can see the splunkd.log are getting indexed in the index="_internal" and I can see the "phoned" log there.

[woodcock]

It is REALLY difficult to diagnose when you have not supplied any configuration file details. Show us the deployment_apps that you created and the contents of each file which MUST include at least inputs.conf and outputs.conf and tell us the path to each.

[mudragada]

Figured it out. There was a rogue tenant.conf file in the deploymentclient - which is used for the forwarding. Once removed, the indexing started to work.

[woodcock]

Wow, I never would have expected that!