Archive
Highlighted

Issue with Show Source when multiple splunk_servers index the same file

Communicator

We have a file being monitored, and the default output is a round-robin to four indexers.
The results show up just fine, but when you click on Show Source for an event, only the events indexed by the same splunk_server are displayed. Is there a way to get Show Source to display all of the events as they originally appeared in the log file, regardless of which server indexed them?

Tags (1)
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Path Finder

Hi, the best way is use load balance in the splunk forwarder instead round-robin.
Try that:

[tcpout:LB_forwarders]
autoLB=true
server=<IP_SERVER_A>:8089,<IP_SERVER_B>:8089,<IP_SERVER_C>:8089,<IP_SERVER_D>:8089
autoLBFrequency=7

[tcpout]
defaultGroup=LB_forwarders
disabled=false
0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Communicator

Sorry if I wasn't clear, but yes, that is what I'm doing.
I think of autoLB as round-robin, but I should have used the proper vernacular.

0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Path Finder

Have you configured the distributed search on your search head ?

0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Communicator

Yes, as I said, the events show up correctly in the event viewer, being pulled in from all indexers.

The issue only shows up when you try to Show Source. In that case, only the events indexed by the same indexer as the selected event appear in the Show Source window. The behavior is somewhat understandable, but not really desirable; the whole point of Show Source is to display the original context of the event.

0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Path Finder

I have almost the same environment as yours and here everything works fine! Any splunk forwarders, four indexers and two search heads; So, that file you are indexing, is it in the same location in all forwarders ?

0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Communicator

To be clear, I am talking about the monitoring of a single file on a single forwarder. As the file grows, the autoLB will switch (every 7 seconds, in your case) which indexer sees chunks of that same file. The distributed search then returns results from all the indexers, but show source on one event in the eventviewer only shows source events from the same indexer that saw the original event.

0 Karma
Highlighted

Re: Issue with Show Source when multiple splunk_servers index the same file

Communicator

In your setup, when you look at one source from one host, how many splunkservers do you see? In the event viewer, pick two adjacent events that are reasonably close in time but have different splunkservers. When you do a Show Source on one of those events, can you see the other event in the resulting log output?

0 Karma