I think I found the issue. We recently updated to Avast Business Pro antivirus. This version has it's own firewall that DID NOT copy existing exceptions from Windows Defender firewall. Creating exceptions in Avast did not resolve the issue. After several posts on the Avast help forum, the easiest (only?) way to fix this is to turn off Avast firewall and revert to Windows Defender firewall.
After doing this, Splunk works as intended until Avast is restarted and automatically re-enables the firewall. To prevent this you must (from the Avast management console) create a new settings template with Avast firewall disabled and apply it to the Splunk server.
Hope this helps others that may run in to this same issue.
Happy Splunking.
Splunk>Your data.
... View more