topic Can you explain Indexer functionality with inputs.conf configured for /var/log/? in Getting Data In

Can you explain Indexer functionality with inputs.conf configured for /var/log/?

kmarciniak — Tue, 29 Sep 2020 23:11:49 GMT

I have Indexers in a cluster running Splunk_TA_nix. I'm monitoring /var/log in inputs.conf. I can see the log events from the search head with a splunk_server from a different Indexer in the cluster. Two questions

1) How did the /var/log/messages, as an example, get indexed? Did it get indexed locally, and if so, how did it know to do that? Or did the events get forwarded to other indexers in the cluster like how our heavy forwarders use Indexer-discovery by contacting the Cluster master for the list of indexers? I ask because I do not see any outputs.conf being configured on Indexers showing any auto-discovery. The cluster master settings are only in the server.conf file. I do not see how these local OS logs are being indexed and it's bothering me.

2) Can I assume the search results showing the /var/log/messages from host1 being seen in the results as splunk_server=host2 is due to replication or is it from host1 forwarding to host2 for indexing?

thanks

Re: Can you explain Indexer functionality with inputs.conf configured for /var/log/?

woodcock — Fri, 08 Feb 2019 22:10:25 GMT

The splunk_server tells you which Indexer handled and stored the incoming event. If the host value for the event is the same, then the event got to the indexer because it was already on the indexer. If the host value is something else, then that server sent the events to the indexer, probably directly (but possibly indirectly); you need to see what is in the outputs.conf files on the host.

Re: Can you explain Indexer functionality with inputs.conf configured for /var/log/?

kmarciniak — Tue, 29 Sep 2020 23:11:52 GMT

1) The key differentiator here is the host is an "indexer" itself. I am monitoring /var/log/* via inputs.conf of the splunk_ta_nix. There are no configuration settings in the indexer's outputs.conf referencing any auto-discovery for its index cluster. So how did /var/log/messages get indexed?
1) For indexers only, does setting an inputs.conf to monitor a file just magically get indexed locally with no outputs.conf file setting showing any destination?
2) For indexers only, does the indexer just know to use auto-discovery since its part of the cluster environment and will then magically look at its server.conf for the CM and get its list of indexers to forward to and perhaps including itself?
3) in my search results the indexer is host1 and the splunk_server was indexer host2 and indexer host3.

I'm still perplexed as to how /var/log/messages from an indexer running splunk_TA_nix is getting indexed.

Re: Can you explain Indexer functionality with inputs.conf configured for /var/log/?

woodcock — Fri, 08 Feb 2019 22:33:52 GMT

Some TAs are created with some settings enabled. It looks like this one is created with some settings in the inputs.conf file enabled. It is easy to check. Because the Indexer will index local files to itself, any inputs.conf that has something that it can find, will be indexed, provided the splunk instance on the indexer has been restarted.

Re: Can you explain Indexer functionality with inputs.conf configured for /var/log/?

kmarciniak — Tue, 29 Sep 2020 23:11:54 GMT

What you just said "Because the Indexer will index local files to itself" is my question. So where is setting to automatically index local files to itself? This was the part i was wondering about. So if it indexes local files to itself such as /var/log/messages from the Splunk_TA_nix where is this setting? Or do you just take it for granted?
Also, if the indexer is indexing its local files from any inputs.conf automatically, then if I run a search for these events in indexer host1 i see splunk_server showing different indexers host2 and host3. Does this mean the indexed data from host1 was replicated over to other indexers and the search just happened to use the data from host2 and host 3 instead of host1?