I have fully configured cluster running Splunk 6.6.5. All indexers and search heads work properly with other inputs. I added new TCP input where 5 workstations send their logs. But I got only one event indexed (from one ws, on one indexer) and new events never show up. I've tested network communication, all ports are open, telnet packets fly smoothly. I also checked if indexers' queues become blocked, but they are not.
What could cause this issue?
have you checked for data in "the past" or "in the furure"? If timestamp recognition goes wrong, data can be indexed at the wrong time
Have you checked, if your one event is only one event? Sometimes if there are wring configs in props.conf. Splunk recognized multiple Events as just one event. Onother problem might be that the forwarder has no read access to the logfile. Some appliances can change access after writing to a file even if you configured the read access for the Splunk OS user.
You are right, we had some issues with encoding and Splunk just dropped events after 10 kb limit was reached.