Missing new indexes on Heavy Forwarder

[Bisho-Fouad]

Hello , i just created new index on cluster master for new integrated log source, but can not find this new index on heavy forwarders to be configured as new data input.

any recommendations for such as situation ?

[Bisho-Fouad]

Hello @VatsalJagani  , as you said no need to create index on all heavy forwarders,

but let me ask something, when i received logs from same new log source, how to differentiate between different logs sources from the same log source? 

[isoutamo]

Usually you are configuring inputs into some own app. Inside this app there is inputs.conf where you have defined needed attributes like sourcetype, source and index where to send events.

Have you already read this https://docs.splunk.com/Documentation/Splunk/latest/Data/Getstartedwithgettingdatain ?

If you are doing regularly indexing and adding new data sources you should participate to System Admin and also Data Admin courses to fully understand the way how this should manage with splunk.

[Bisho-Fouad]

Hello Again, I created new index on one of heavy forwarders, how to make this index reflect in all other heavy forwarders ?

[VatsalJagani]

@Bisho-Fouad - Why do you want to create input on all heavy forwarders? 

* I think this will duplicate the data.

There is no necessity to create index on all heavy forwarders. Only where you are configuring the input.

[VatsalJagani]

@Bisho-Fouad - The short answer is to create the same index on the HF.

(It will not be used for storing data, the purpose is just that you will see the index name while configuring the inputs.)

I hope this helps!!! Kindly upvote if it does!!!

[isoutamo]

Hi

when you are creating a new index on Cluster Master, it just create that on cluster peers not anywhere else. If/when you want that index definition also on HF you must add it there manually. Based on your environment there are couple of things to remember and you must modify those when you are adding indexes on HF and/or your SH(s).

• if you are using volumes that must update that to point correct volume definition and/or add "dummy" volume definition to other host
• when you are using non cluster environment on other nodes you should update repFactor from auto to 0

My proposal is that you have separate definitions for volumes and repFactor as default values for cluster and other nodes. Then own file/TA/SA for real index definitions. In that way you could use same index definition files on all nodes instead of update it every time. Just store it to git and then deploy it from there. And of course  you must deploy that another app/files which contains specific definitions once before real index definitions can deployed. If you forget this then your nodes didn't start as they haven't have valid index definitions.

r. Ismo