We have a file being monitored, and the default output is a round-robin to four indexers.
The results show up just fine, but when you click on Show Source for an event, only the events indexed by the same splunk_server are displayed. Is there a way to get Show Source to display all of the events as they originally appeared in the log file, regardless of which server indexed them?
Hi, the best way is use load balance in the splunk forwarder instead round-robin.
[tcpout:LB_forwarders] autoLB=true server=<IP_SERVER_A>:8089,<IP_SERVER_B>:8089,<IP_SERVER_C>:8089,<IP_SERVER_D>:8089 autoLBFrequency=7 [tcpout] defaultGroup=LB_forwarders disabled=false
Sorry if I wasn't clear, but yes, that is what I'm doing.
I think of autoLB as round-robin, but I should have used the proper vernacular.
Yes, as I said, the events show up correctly in the event viewer, being pulled in from all indexers.
The issue only shows up when you try to Show Source. In that case, only the events indexed by the same indexer as the selected event appear in the Show Source window. The behavior is somewhat understandable, but not really desirable; the whole point of Show Source is to display the original context of the event.
I have almost the same environment as yours and here everything works fine! Any splunk forwarders, four indexers and two search heads; So, that file you are indexing, is it in the same location in all forwarders ?
To be clear, I am talking about the monitoring of a single file on a single forwarder. As the file grows, the autoLB will switch (every 7 seconds, in your case) which indexer sees chunks of that same file. The distributed search then returns results from all the indexers, but show source on one event in the eventviewer only shows source events from the same indexer that saw the original event.
In your setup, when you look at one source from one host, how many splunkservers do you see? In the event viewer, pick two adjacent events that are reasonably close in time but have different splunkservers. When you do a Show Source on one of those events, can you see the other event in the resulting log output?