I set up a Linux forwarder to forward os logs to a Windows indexer as a test. The Windows indexer is running Splunk Free (500MB per day license limit).
After setting this up I am only seeing Splunk internal logs (e.g. _internal) when querying data on this indexer. The index (index=myindex) on the indexer which I have configured to receive the os logs is empty. Nothing has been written to this index. The forwarder has been configured to forward data to index=myindex.
Any ideas why my indexer is indexing internal logs from this forwarder but is not indexing os logs?
This forwarder is also forwarding the os logs to a separate Linux Splunk instance and it is working as expected there (i.e. This Linux Splunk instance indexes both os and Splunk internal logs).
Receiving internal logs means that you correctly configured connection between UFs and Indexer, but probably there's an input error: what did you used on Universal Forwarder for input: TAnix or a manual inputs.conf?
Then which index did you configured on inputs.conf in UFs?
If you used TA_nix, by default it sends unix logs to os index.
The index=index_names in inputs.conf on the Linux forwarder match the indexes created on the Windows indexer where the data is not being indexed. Yet this Windows indexer is having no issue indexing _internal logs from the Linux forwarder.
It is a manual inputs.conf on the forwarder. Both the _internal and os logs get indexed fine in the separate Linux Splunk instance but the Windows indexer only chooses to index the _internal logs. Both the Windows indexer and the separate Linux Splunk instance have the same indexes created.
This is weird
1: You did not setup your
inputs.conf correctly on your forwarder. If so, there should be errors in
_internal from your forwarder that indicate this.
2: You did not setup
myindex correctly on your indexer. If so, there should be errors in
_internal from your indexer that indicate this.
The issue was with outputs.conf.
There were a number of outputs.conf configuration with similar tcpout stanzas which were causing some config to take precedence over other configuration.