Getting Data In

Do sourcetypes and indexes created in a Heavy Forwader replicate onto Indexers and Search Heads?

jo54
Loves-to-Learn Lots

In my HF, I parsed an example log from a local file and stored the parsing as a sourcetype. Then, I created an index for the events storage. 

However, I cannot locate this in the Search Head after searching for it. I am uncertain as to whether or not the index exists in the indexer.

There are two HFs, two Indexers, two Search Heads, and one Cluster Master/Deployment Server/ License Manager. The outputs.conf should be properly set, as more integrations are already operational.

0 Karma

PickleRick
Ultra Champion

In general, you can divide Splunk infrastructure into three layers.

- Search Heads - if clustered they replicate some configuration between them within the cluster. Otherwise you use deployer to push configuration to them

- Indexers - you use cluster master to push configuration

- Forwarders - you use deployment server (sometimes you use deployment server to push configuration from DS to single search-heads or single indexers but that's a very unusual situation).

There is no replication between layers as such. You could push from DS to deployer or cluster master but that's again - a very unusual situation.

To make long story short - if you push something to a forwarder, it stays there. If you push something to search-heads, it's on search heads. And so on - you manage configuration on each layer separately.

Oh, and you don't create indexes on HFs. Since HF only processes events and forwards events to indexers - you don't store events locally - you don't need indexes on HF.

0 Karma

gcusello
Legend

Hi @jo54,

all the conf files are distributed in diferent way between different roles:

  • on heavy and Universal Forwarders are deployed by Deployment Server,
  • on Search Head Clusters are deployed by Deployer,
  • On Indexer Cluster are deployed by Master Node

There isn't any other cross replication.

You eventually can use Deployment Server to deploy apps to the Master Node and it deploys to Indexers but not directly.

Anyway, usually indexes.conf are only on Indexers, not on SHs or HFs.

Then isn't a good idea to have Master Node and Deployment Server on the same machine if you have to manage more than 50 clients or you have manu data to index.

And anyway, even if one machine for the same role, you have to use two different mechanism to deploy apps, so it isn't possible to syncronize them and especially it isn't requested because indexes.conf usually is on Indexers.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Knowledge objects, including indexes and sourcetypes, created on an HF or indexer do not propagate to other instances.  A KO created on a search head in a SHC will propagate to other members of the cluster, but that is the only case where that happens.

If you did not create the index on the indexer then the index does not exist.  Since you have an indexer cluster, new indexes should be defined in an indexes.conf file in the master-apps (manager-apps in Splunk 9) directory of the Cluster Manager.  The index will be created after the cluster bundle is applied to the indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

jo54
Loves-to-Learn Lots

Thanks a lot. I will definitely try this. I guess this can result in a rolling-restart scenario. Would the sourcetype be also synchronised?

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...