Splunk Search

Issue with Show Source when multiple splunk_servers index the same file

mslvrstn
Communicator

We have a file being monitored, and the default output is a round-robin to four indexers.
The results show up just fine, but when you click on Show Source for an event, only the events indexed by the same splunk_server are displayed. Is there a way to get Show Source to display all of the events as they originally appeared in the log file, regardless of which server indexed them?

Tags (1)

mamaral
Path Finder

I have almost the same environment as yours and here everything works fine! Any splunk forwarders, four indexers and two search heads; So, that file you are indexing, is it in the same location in all forwarders ?

0 Karma

mslvrstn
Communicator

In your setup, when you look at one source from one host, how many splunk_servers do you see? In the event viewer, pick two adjacent events that are reasonably close in time but have different splunk_servers. When you do a Show Source on one of those events, can you see the other event in the resulting log output?

0 Karma

mslvrstn
Communicator

To be clear, I am talking about the monitoring of a single file on a single forwarder. As the file grows, the autoLB will switch (every 7 seconds, in your case) which indexer sees chunks of that same file. The distributed search then returns results from all the indexers, but show source on one event in the eventviewer only shows source events from the same indexer that saw the original event.

0 Karma

mamaral
Path Finder

Have you configured the distributed search on your search head ?

0 Karma

mslvrstn
Communicator

Yes, as I said, the events show up correctly in the event viewer, being pulled in from all indexers.

The issue only shows up when you try to Show Source. In that case, only the events indexed by the same indexer as the selected event appear in the Show Source window. The behavior is somewhat understandable, but not really desirable; the whole point of Show Source is to display the original context of the event.

0 Karma

mamaral
Path Finder

Hi, the best way is use load balance in the splunk forwarder instead round-robin.
Try that:

[tcpout:LB_forwarders]
autoLB=true
server=<IP_SERVER_A>:8089,<IP_SERVER_B>:8089,<IP_SERVER_C>:8089,<IP_SERVER_D>:8089
autoLBFrequency=7

[tcpout]
defaultGroup=LB_forwarders
disabled=false
0 Karma

mslvrstn
Communicator

Sorry if I wasn't clear, but yes, that is what I'm doing.
I think of autoLB as round-robin, but I should have used the proper vernacular.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...