I ran into this problem.
I installed the DCN, the connections all checked out green, and I was ready to go. I did a search and my VMwre app dashboard all came up with data. Brillant, so far.
The Data Collection Node (DCN) that comes with Splunk has a 5GB disk. The default for for the dispatcher for doing searches is 5GB. What happened to me is that the VMware app came up, populated the dashboards, and then never collected another thing. The reason, revealed by tailing the splunkd.log file on the DCN is that there was not enough space on the virtual disk drive on the DCN VM. I solved it by having the VM admin up the space available to $SPLUNK_HOME directory on the DCN. Ultimately, I rolled my own DCN because the VMWare schema couldn't (or wouldn't) grow the VM directory. Splunk was in /home/splunk vice /opt/splunk. oh well.
The other possible solution is to change the minimum disk space required for the dispatcher in Splunk when conduction searches. You could lower it to 2 GB and then start getting search data back. If you problem is similar to the one I encountered, this might help.
... View more