We have set up "Splunk Forwarder Management" and apps are being successfully deployed to the clients that are polling the Splunk server on port 8089.
We have ensured the check-mark "Restart Splunkd" is checked for the apps being deployed. But strange this is the check-mark gets automatically unchecked. Not sure if this has anything to do with the problem, but logs stop getting forwarded from the clients and when we do "splunk list forward-server", we see that the splunk forwarder is marked "Configured but inactive forwards:"
After running "splunk restart" everything gets back to normal.
Question: What causes the forwarder to become inactive and stop forwarding logs?
When you use the splunk list forward-server is only shows you forward servers that are active in the instant you ran the command, so unless you are actively pushing data, none of your forward servers are going to show up as being "active". Once you are pushing data, you can also incorporate the watch command to auto refresh the command and catch an active forward: watch -n 5 splunk list forward-server,Only posting this because none of the above options helped me discover that my config was fine:
A forward server will only show up as "active" using splunk list forward-server command if it is actively forwarding data. Run the command a few times and chances are you will see it move to the active state. You can also incorporate the watch command to auto refresh the command: watch -n 5 splunk list forward-server