Forwarding events to custom index on Cluster peers

[Mag2sub]

My cluster peers have a cluster app with indexes.conf which have custom indexes defined

Clustre bundle has been deployed and works perfectly slave apps have artifacts...receiving also has been setup in peers

My forwarder is correctly configured to send events to cluster peers with ACK

However whenever i try add monitors(on forwarder) sources to specific indexes which exist on the receiving cluster peers ..get error in "handler monitor" paramter index no currently active index..not loaded,disabled,or not defined

The index is however available in the receiving cluster peers

Please suggest ...thanks

[jkerai · Accepted solution]

The only issue was with HWF CLI validating that you have input configured to valid index that is present on HWF instance. Under the cover, CLI validates index, creates inputs.conf stanza with desired index setting and reloads inputs.conf.

So, the workarounds are, to repeat 🙂
1) Either define the index to make validation code happy. This allows CLI to continue with creating inputs stanza with desried index setting.

2) or just edit inputs.conf directly specifying the index(but not defining them in HWF, skipping index validation step) and ask splunk explicitly to reload inputs.conf with REST endpoint at https://splunk-server:8089/services/data/inputs/monitor/_reload.

UF does not perform index validation, as it does not data locally. So, just adding monitor with desired index would work fine.

[yannK]

fixed in splunk 6.*

./splunk version
Splunk Universal Forwarder 6.0.2 (build 196940)

./splunk add monitor "/path/to/my/feed" -index doesnotexist
Added monitor of '/path/to/my/feed'.

./splunk cmd btool inputs list
[monitor:////path/to/my/feed]
_rcvbuf = 1572864
disabled = false
host = myhost
index = doesnotexist

[Mag2sub]

Hmmm...just clarifying further or splitting the hair as it seems 🙂 ..or maybe need some jolt of java
ONLY after adding the custom index on my forwarder was an issue with HWF for this to work ? with receiving cluster peers

and in case i go for UF ...i dont need to add the custom index at UF level ..all i need to do is add monitor i shuld be done

really appreciate!

[jkerai]

Unless there is a very specific reason to use regular/heavy weight forwarder(HWF), I would recommend using Universal forwarder(UF). If you use UF, you will not hit this bug. If you edit inputs.conf manually(you will need to for HWF for this case), you will need to either restart forwarder, or hit REST endpoint https://splunk-server:8089/services/data/inputs/monitor/_reload to ask splunkd to reload. Adding inputs via monitor is definitely simpler.

[Mag2sub]

Hmm..bear with me 🙂

1)Normally i would not need to add indexes on the forwarder i would just add inputs

But in this case only after adding indexes on forwarder it was forwarded to the receiver index...

so are you recommending i use a universal forwarder and i will not run into this bug at all ? and just be adding inputs through add monitor command or directly edit inputs.conf it should be ok on a universal fowarder

Thanks!

[jkerai]

1. If you are using Universal forwarder, then CLI would not throw the error. Seems like you are using regular/heavy weight forwarder, so CLI incorrectly validates and errors out. I would recommend just editing inputs.conf and specifying index. This way you can avoid adding indexes on forwarder. This is not yet addressed in 6.0.

2. Have not seen any issue specific to authentication on search head. May want to get in touch with support and upload diag. What is the specific error message reported by search head?

[Mag2sub]

1 Should we be adding these custom indexes normally to a forwarder...if its a known issue with Cluster based fowarding is there a known bugid for this somewhere and is it resolved in splunk 6 ? or splunk 5.0.5?

2)Run into authetication issues at search head now ..ugh...autentication issues sis very flaky...restart or readd serach head resolves everytime hmmm confidence just dropping

Appreciate inputs

[jkerai]

I meant you to add custom indexes on forwarder that were pushed out from master to cluster peers.

As far as data not found on cluster search head is concerned, I think those were already forwarded to main index in your previous attempt. Could you check if they are in main index?

[Mag2sub]

I added indexes my custom indexes and also idx1 on forwarder and restarted the add monitor now does not fail BUT do not see nay data on the cluster search head for those indexes ...howoever the main was forwarded correctly

Appreciate pointers..did you really mean for me to add a index name idx1 on fowarder ? or only my custom indexes?

[jkerai]

Seems like you are using Heavy weight forwarder(HWF).

Please define the index on the forwarder instance as well. The index on HWF would not take any space, but will merely allow validation to pass.
Or,
You can edit inputs.conf to add 'index=' to specific input stanza.

For Universal forwarder, we skip validation of index, so CLI with index param does not throw any error.

[jkerai]

Seems like CLI/REST endpoint is running into validation error. You can try one of the following:

• You may want to try adding this index (splunk add index -name 'idx1') on the forwarder as well to avoid this error.

• Edit inputs.conf and for monitor stanza add 'index=idx1' param and restart. Forwarding should happen to correct index.

I will file a ticket for CLI/REST issue.

[jtworzydlo]

I would try to forward the data to the main index to double check the communication between the forwarder and the peers.
I would also go to the UI of every peer to Manager->Indexes and take a look if the index is visible there, enabled, and the configuration of it works fine.
You might use this:

| eventcount summarize=false index=* |
dedup index | fields index

to see if you index is seen by the search head.

[Mag2sub]

yes i had checked the same after deploying the cluster app artifacts indexes and inputs and the search head does return the custom indexes correctly and the indexes are alive and enabled in the peer nodes

I have also checked by forwrading to main ..the add monitor works ....however it fails in adding monitor for the custom indexes

Do we need create the custom indexes on the forwarder too ?
I have never known that to be case as the index always needs to be created on the target indexer

Confused