How to replicate a kvstore from a Splunk HF to the SHC?

[Glasses2]

I am reviewing a previously created lookup that is based on a KV-store collection.

There is a custom script (contained in a custom kvstore app) on an HF that pulls data into a file.csv, processes the file changes, and then updates a kvstore collection.   

My question is "how do I verify this collection (i.e. FooBar) is being replicated to the Search Head Cluster?"

The collections.conf on the HF shows 

[FooBar]

replicate=true 
field.<something1> =string 

field.<something2> =string

field.<something3> =string 

The same collections.conf is on the SHC (in /opt/splunk/etc/apps/kv_store_app/local ) probably created via the WebUI lookup setting page... it says only 

[FooBar]

disabled=0

when I run  " | inputlookup FooBar " on both HF and the SHC members, the results are different, so appears to be out of sync or broken.

Any advise or references appreciated for this scenario.

Thank you

[PickleRick]

As @livehybrid already pointed out - there is only replication of lookups (either kvstore-backed or csv-backed within the SHC). The contents of a particular lookup can be sent as part of a knowledge bundle to indexer(s) if they are needed for a search. But that's it.

If an app on your HF uses kvstore (IIRC some modular inputs do so to store "state"), that instance is completely stand-alone.

Depending on your needs there might be some way to "replicate" the contents but it would probably mean treating your HF as SH, spawning a search which would effectively do something like

| inputlookup <...> | collect <...>

Events created this way would get forwarded to your indexer(s). And on your SHC you'd have to schedule a search which would do the opposite operation - search for the latest indexed events and based on them do outputlookup.

[livehybrid · Accepted solution]

Hi @Glasses2 

I think there may be some confusion here, the replication of a KV Store collection is only between members of a searchhead cluster (SHC) - it isnt possible to natively replicate KV Stores from a HF to SHC, this would need further architecting with additional scripts (there may be apps which can do this?) to allow this kind of replication to occur. 

Check out the "KV Store Tools Redux" app (https://splunkbase.splunk.com/app/5328) as this has the ability to push KV stores to remote instances and might solve your requirement.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[Glasses2]

@livehybrid 
Thank you for the sanity check, I was not understanding how this would have worked, I was not familiar if that sort of replication was even possible (HF to SHC).

Apparently some other mechanism (to transfer the file.csv to the SHC) is missing, broken, or unknown at this time.

I will keep you posted

Thanks

[Glasses2]

there was another script on the HF that updated the kvstore on the shc... so that was the mystery

Thanks