Splunk Enterprise Setup

[Twagner79]

Hey all, I am stumped and need some help, I am configuring a system stack with Splunk enterprise on it. It is relatively small, only 11 systems. I have the web interface installed with a license, forwarders and apps pushed out to systems, and port listeners open on 9997 for the forwarder to talk back to in the forwarding and receiving tab. I know there is some communication because I can see all of the systems in the forwarder management tab, however I cannot get any data into our dashboards. The only system data I can find and search is that of the server where the main instance is located. I have indexes made for all the different types of data, (linux_audit, Win_security, ETC). No data from the forwarders themselves is coming through. My only other thought is a firewall issue and that the correct port isn't open but beyond that I had no idea.  I am sorry for the ignorance, this is my first real time setting this up and the Splunk documentation isn't super helpful for troubleshooting. Thanks in advance! 

[isoutamo]

Can you see those nodes on MC’s forwarders? This needs enabling forwarder monitoring on mc - setup tab.

Another option is do a query 

index=_internal | stats count by host

That query shows if those nodes can send their internal logs to splunk.

You should check that your server haven’t blocked port 9997/tcp by host based firewall. Open that port if needed. Also ensure that you have ticked “reboot” box on fwd management. Otherwise splunk just install those configurations to UFs, but don’t take those to use with rebooting UF.

r. Ismo