Issue with Show Source when multiple splunk_servers index the same file

[mslvrstn]

We have a file being monitored, and the default output is a round-robin to four indexers.
The results show up just fine, but when you click on Show Source for an event, only the events indexed by the same splunk_server are displayed. Is there a way to get Show Source to display all of the events as they originally appeared in the log file, regardless of which server indexed them?

[mamaral]

I have almost the same environment as yours and here everything works fine! Any splunk forwarders, four indexers and two search heads; So, that file you are indexing, is it in the same location in all forwarders ?

[mslvrstn]

In your setup, when you look at one source from one host, how many splunk_servers do you see? In the event viewer, pick two adjacent events that are reasonably close in time but have different splunk_servers. When you do a Show Source on one of those events, can you see the other event in the resulting log output?

[mslvrstn]

To be clear, I am talking about the monitoring of a single file on a single forwarder. As the file grows, the autoLB will switch (every 7 seconds, in your case) which indexer sees chunks of that same file. The distributed search then returns results from all the indexers, but show source on one event in the eventviewer only shows source events from the same indexer that saw the original event.

[mamaral]

Have you configured the distributed search on your search head ?

[mslvrstn]

Yes, as I said, the events show up correctly in the event viewer, being pulled in from all indexers.

The issue only shows up when you try to Show Source. In that case, only the events indexed by the same indexer as the selected event appear in the Show Source window. The behavior is somewhat understandable, but not really desirable; the whole point of Show Source is to display the original context of the event.

[mamaral]

Hi, the best way is use load balance in the splunk forwarder instead round-robin.
Try that:

[tcpout:LB_forwarders]
autoLB=true
server=<IP_SERVER_A>:8089,<IP_SERVER_B>:8089,<IP_SERVER_C>:8089,<IP_SERVER_D>:8089
autoLBFrequency=7

[tcpout]
defaultGroup=LB_forwarders
disabled=false

[mslvrstn]

Sorry if I wasn't clear, but yes, that is what I'm doing.
I think of autoLB as round-robin, but I should have used the proper vernacular.