Deployment Architecture

Forwarding events to custom index on Cluster peers

Mag2sub
Path Finder

My cluster peers have a cluster app with indexes.conf which have custom indexes defined

Clustre bundle has been deployed and works perfectly slave apps have artifacts...receiving also has been setup in peers

My forwarder is correctly configured to send events to cluster peers with ACK

However whenever i try add monitors(on forwarder) sources to specific indexes which exist on the receiving cluster peers ..get error in "handler monitor" paramter index no currently active index..not loaded,disabled,or not defined

The index is however available in the receiving cluster peers

Please suggest ...thanks

Tags (1)
0 Karma
1 Solution

jkerai
Splunk Employee
Splunk Employee

The only issue was with HWF CLI validating that you have input configured to valid index that is present on HWF instance. Under the cover, CLI validates index, creates inputs.conf stanza with desired index setting and reloads inputs.conf.

So, the workarounds are, to repeat 🙂
1) Either define the index to make validation code happy. This allows CLI to continue with creating inputs stanza with desried index setting.

2) or just edit inputs.conf directly specifying the index(but not defining them in HWF, skipping index validation step) and ask splunk explicitly to reload inputs.conf with REST endpoint at https://splunk-server:8089/services/data/inputs/monitor/_reload.

UF does not perform index validation, as it does not data locally. So, just adding monitor with desired index would work fine.

View solution in original post

0 Karma

jkerai
Splunk Employee
Splunk Employee

The only issue was with HWF CLI validating that you have input configured to valid index that is present on HWF instance. Under the cover, CLI validates index, creates inputs.conf stanza with desired index setting and reloads inputs.conf.

So, the workarounds are, to repeat 🙂
1) Either define the index to make validation code happy. This allows CLI to continue with creating inputs stanza with desried index setting.

2) or just edit inputs.conf directly specifying the index(but not defining them in HWF, skipping index validation step) and ask splunk explicitly to reload inputs.conf with REST endpoint at https://splunk-server:8089/services/data/inputs/monitor/_reload.

UF does not perform index validation, as it does not data locally. So, just adding monitor with desired index would work fine.

0 Karma

yannK
Splunk Employee
Splunk Employee

fixed in splunk 6.*

./splunk version
Splunk Universal Forwarder 6.0.2 (build 196940)

./splunk add monitor "/path/to/my/feed" -index doesnotexist
Added monitor of '/path/to/my/feed'.

./splunk cmd btool inputs list
[monitor:////path/to/my/feed]
_rcvbuf = 1572864
disabled = false
host = myhost
index = doesnotexist

0 Karma

Mag2sub
Path Finder

Hmmm...just clarifying further or splitting the hair as it seems 🙂 ..or maybe need some jolt of java
ONLY after adding the custom index on my forwarder was an issue with HWF for this to work ? with receiving cluster peers

and in case i go for UF ...i dont need to add the custom index at UF level ..all i need to do is add monitor i shuld be done

really appreciate!

0 Karma

jkerai
Splunk Employee
Splunk Employee

Unless there is a very specific reason to use regular/heavy weight forwarder(HWF), I would recommend using Universal forwarder(UF). If you use UF, you will not hit this bug. If you edit inputs.conf manually(you will need to for HWF for this case), you will need to either restart forwarder, or hit REST endpoint https://splunk-server:8089/services/data/inputs/monitor/_reload to ask splunkd to reload. Adding inputs via monitor is definitely simpler.

0 Karma

Mag2sub
Path Finder

Hmm..bear with me 🙂

1)Normally i would not need to add indexes on the forwarder i would just add inputs

But in this case only after adding indexes on forwarder it was forwarded to the receiver index...

so are you recommending i use a universal forwarder and i will not run into this bug at all ? and just be adding inputs through add monitor command or directly edit inputs.conf it should be ok on a universal fowarder

Thanks!

0 Karma

jkerai
Splunk Employee
Splunk Employee
  1. If you are using Universal forwarder, then CLI would not throw the error. Seems like you are using regular/heavy weight forwarder, so CLI incorrectly validates and errors out. I would recommend just editing inputs.conf and specifying index. This way you can avoid adding indexes on forwarder. This is not yet addressed in 6.0.

  2. Have not seen any issue specific to authentication on search head. May want to get in touch with support and upload diag. What is the specific error message reported by search head?

0 Karma

Mag2sub
Path Finder

1 Should we be adding these custom indexes normally to a forwarder...if its a known issue with Cluster based fowarding is there a known bugid for this somewhere and is it resolved in splunk 6 ? or splunk 5.0.5?

2)Run into authetication issues at search head now ..ugh...autentication issues sis very flaky...restart or readd serach head resolves everytime hmmm confidence just dropping

Appreciate inputs

0 Karma

jkerai
Splunk Employee
Splunk Employee

I meant you to add custom indexes on forwarder that were pushed out from master to cluster peers.

As far as data not found on cluster search head is concerned, I think those were already forwarded to main index in your previous attempt. Could you check if they are in main index?

0 Karma

Mag2sub
Path Finder

I added indexes my custom indexes and also idx1 on forwarder and restarted the add monitor now does not fail BUT do not see nay data on the cluster search head for those indexes ...howoever the main was forwarded correctly

Appreciate pointers..did you really mean for me to add a index name idx1 on fowarder ? or only my custom indexes?

0 Karma

jkerai
Splunk Employee
Splunk Employee

Seems like you are using Heavy weight forwarder(HWF).

Please define the index on the forwarder instance as well. The index on HWF would not take any space, but will merely allow validation to pass.
Or,
You can edit inputs.conf to add 'index=' to specific input stanza.

For Universal forwarder, we skip validation of index, so CLI with index param does not throw any error.

0 Karma

jkerai
Splunk Employee
Splunk Employee

Seems like CLI/REST endpoint is running into validation error. You can try one of the following:

  • You may want to try adding this index (splunk add index -name 'idx1') on the forwarder as well to avoid this error.

  • Edit inputs.conf and for monitor stanza add 'index=idx1' param and restart. Forwarding should happen to correct index.

I will file a ticket for CLI/REST issue.

0 Karma

jtworzydlo
Path Finder

I would try to forward the data to the main index to double check the communication between the forwarder and the peers.
I would also go to the UI of every peer to Manager->Indexes and take a look if the index is visible there, enabled, and the configuration of it works fine.
You might use this:

| eventcount summarize=false index=* |
dedup index | fields index

to see if you index is seen by the search head.

0 Karma

Mag2sub
Path Finder

yes i had checked the same after deploying the cluster app artifacts indexes and inputs and the search head does return the custom indexes correctly and the indexes are alive and enabled in the peer nodes

I have also checked by forwrading to main ..the add monitor works ....however it fails in adding monitor for the custom indexes

Do we need create the custom indexes on the forwarder too ?
I have never known that to be case as the index always needs to be created on the target indexer

Confused

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...