Getting Data In

Splunk UF clients phoned but are not indexing logs

mudragada
Path Finder

I've deployed the deployment-app on the deployment client from deployment server.

The server appeared on the phoned list. But is not indexing the logs.

The splunkforwarder logs don't show any error and the splunk user has the permissions to read the logs.
Where can I check what is not causing the indexing?

0 Karma
1 Solution

woodcock
Esteemed Legend

Did you deploy an app that contains an inputs.conf? Did you configure the forwarder to point to an indexer (or deploy an app with outputs.conf)? Did you set the restart splunkd bit on the app that contains the inputs.conf file (or restart splunk on the forwarder)?

View solution in original post

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Did you specify and inputs.conf and outputs.conf in your deployment-apps?

Login to one of your forwarders where this was deployed on. $SPLUNK_HOME/etc/apps/<YOUR_APP>/local

The inputs.conf file should be pointing to a log file which is sitting on the machine. It should also specify the index and sourcetype. Next inspect the outputs.conf file and verify its pointing to an indexer(s). It looks like you've already determined that there is an open port between forwarder and indexer aswell.

Lastly, you need to restart the forwarders for the new conf changes to take affect

0 Karma

mudragada
Path Finder

Yes, checked that. it has the inputs.conf as well and outputs.conf is pointing to the indexer.
And the ports are open - there isn't a read error or connection refused in the splunkd logs.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Did you restart the Splunk Forwarder service after making these changes?

$SPLUNK_HOME/bin/splunk restart

0 Karma

mudragada
Path Finder

Did the restart. Still didn't. At this point I'm looking at, how I can debug the scenario?

0 Karma

masonmorales
Influencer

Does the data show up if you search "all time" against the index? If it does, you may have future-dated data, which would be a parsing issue. (i.e. TZ in props.conf)

0 Karma

woodcock
Esteemed Legend

Did you deploy an app that contains an inputs.conf? Did you configure the forwarder to point to an indexer (or deploy an app with outputs.conf)? Did you set the restart splunkd bit on the app that contains the inputs.conf file (or restart splunk on the forwarder)?

View solution in original post

0 Karma

mudragada
Path Finder

Yes, that's done. I find the app in the /opt/splunkforwarder with the right hostname, index name and all in the inputs.conf. I'm still not seeing the logs. I can see the splunkd.log are getting indexed in the index="_internal" and I can see the "phoned" log there.

0 Karma

woodcock
Esteemed Legend

It is REALLY difficult to diagnose when you have not supplied any configuration file details. Show us the deployment_apps that you created and the contents of each file which MUST include at least inputs.conf and outputs.conf and tell us the path to each.

0 Karma

mudragada
Path Finder

Figured it out. There was a rogue tenant.conf file in the deploymentclient - which is used for the forwarding. Once removed, the indexing started to work.

woodcock
Esteemed Legend

Wow, I never would have expected that!

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!