Getting Data In

Indexer receives forwarded Splunk internal logs but does not receive non-Splunk os events

Motivator

Hi,

I set up a Linux forwarder to forward os logs to a Windows indexer as a test. The Windows indexer is running Splunk Free (500MB per day license limit).

After setting this up I am only seeing Splunk internal logs (e.g. _internal) when querying data on this indexer. The index (index=myindex) on the indexer which I have configured to receive the os logs is empty. Nothing has been written to this index. The forwarder has been configured to forward data to index=myindex.

Any ideas why my indexer is indexing internal logs from this forwarder but is not indexing os logs?

This forwarder is also forwarding the os logs to a separate Linux Splunk instance and it is working as expected there (i.e. This Linux Splunk instance indexes both os and Splunk internal logs).

Thanks

Tags (2)
0 Karma
1 Solution

Motivator

The issue was with outputs.conf.

There were a number of outputs.conf configuration with similar tcpout stanzas which were causing some config to take precedence over other configuration.

View solution in original post

0 Karma

Motivator

The issue was with outputs.conf.

There were a number of outputs.conf configuration with similar tcpout stanzas which were causing some config to take precedence over other configuration.

View solution in original post

0 Karma

Esteemed Legend

Either:
1: You did not setup your inputs.conf correctly on your forwarder. If so, there should be errors in _internal from your forwarder that indicate this.
2: You did not setup myindex correctly on your indexer. If so, there should be errors in _internal from your indexer that indicate this.

0 Karma

Motivator

Both of these were setup correctly. I have fixed the issue.

0 Karma

SplunkTrust
SplunkTrust

Hi Ant1D,
Receiving _internal logs means that you correctly configured connection between UFs and Indexer, but probably there's an input error: what did you used on Universal Forwarder for input: TA_nix or a manual inputs.conf?
Then which index did you configured on inputs.conf in UFs?
If you used TA_nix, by default it sends unix logs to os index.

Ciao.
Giuseppe

0 Karma

Motivator

The index=index_names in inputs.conf on the Linux forwarder match the indexes created on the Windows indexer where the data is not being indexed. Yet this Windows indexer is having no issue indexing _internal logs from the Linux forwarder.

It is a manual inputs.conf on the forwarder. Both the _internal and os logs get indexed fine in the separate Linux Splunk instance but the Windows indexer only chooses to index the _internal logs. Both the Windows indexer and the separate Linux Splunk instance have the same indexes created.

This is weird

0 Karma

SplunkTrust
SplunkTrust

Hi Ant1D,
could you share your inputs.conf?
Ciao.
Giuseppe

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!