Why is my forwarding configuration not forwarding data?

[antifreke]

Good afternoon, working on setting up the final piece of Splunk infrastructure and I have come across a little speed bump. The design is simple, Single Search Head, Dual Indexers, Syslog server (RH). I have data moving from the Syslog server to both indexers, but not from the indexers to the Search Head. My outputs on the indexer are as follows:

[tcpout]
defaultGroup = my_searchhead
indexAndForward = true
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.whitelist = network1
forwardedindex.2.whitelist = guestnetwork
forwardedindex.3.whitelist = network2
forwardedindex.filter.disabled = true

[tcpout:my_searchhead]
server = x.x.x.x:9998

Search head Inputs has the following:

[splunktcp://9998]

Do I need to tweak anything else? Am I missing something really simple and over complicating it?

[lguinn2 · Accepted solution]

You do not send events from your indexers to your search head! You do not need any outputs.conf on the indexers!

Instead, you need to tell the search head where to search. You can do this in the GUI by setting up Distributed Search, or by creating/editing distsearch.conf on your search head. Here is the documentation for adding search peers to the search head.

[antifreke]

So, should I return the outputs.conf files in the indexers back to their default?

I have the distributed search setup on the Search Head. I think I was looking at this architecture completely wrong and misunderstanding the data flow.

I looked at my searc head and did a query for index=network and it works. I was looking at this entire thing backwards. Thank you!!

[lguinn2]

You can return the outputs.conf on the indexers back to what it was before - but usually, the indexers don't need outputs.conf at all...

[jmaple]

This configuration implies you are trying to index your events on your search head as well as your indexer. This is not how Splunk is intended to be used and I'm sure that's not what you intended either.

To search the data indexed on your indexers, simply make your indexers search peers of your search head using distsearch.conf

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Distsearchconf

[jmaple]

This configuration implies that you are indexing events on your search head. Is that your intention?

[suarezry]

Typically, you configure your search head to forward searches to the indexers. You do not actually forward the data from the indexers to the search head.

[antifreke]

So the data stays on the indexers, and the search head performs searches from there to the indexers?