Our splunk instance seems to be indexing logs, but they are not showing up in the search feature. (tcpdump verifies sending and receiving, and useACK in outputs.conf doesn't resend the logs). The index within the web-gui is showing 0MB of data. The _internal index verifies that the connection is made, and bytes of the license are being used. We are running on 6.0.2 on the server and forwarder, and other hosts in other indexes are reporting without an issue. We are using an NFS mount for log storage, and all of the indexes reside on that mount. nfsstats shows 20% r/w.
try the search with index=*
Are you sure the index is in your default search list (see role definitions and their default searched indexes, and users and role associations). If you expressly specify the index you expect to contain the data "index=xyz" as a sub-clause of your basic search do the details appear? If you have a multi-indexer or dedicated searc/indexer deployment, are you sure that the forwarder is forwading to the indexer you expect and that your search head is also searching the correct indexer? When you check your index status page is that on the search head, or on the targetted indexer? Which of your indexes is growing (or last updated).
You should check your main index to make sure they are not going in there.
Well, you are aware that NFS is really recommended against (though mostly for performance reasons)?
Could it be that your logs are older than the configured retention time, i.e. the timestamps (or the parsing of them) will move them straight to frozen (i.e. deleted)?
Grab the SoS app and take a look at the indexing dashboards - those give you an insight into how much data is ending up where.