topic Re: KV Store on Heavy Forwarder in Knowledge Management

KV Store on Heavy Forwarder

rajrsplunk — Fri, 27 Jul 2018 22:25:44 GMT

EDITED:
I am building a TA. I have installed it on my Heavy Forwarder, it writes events to the Indexer.
The TA uses custom python code to extract the data from APIs (http GET calls to my webservice).
I'm using the KVStore to store state of the TA, I need to remember what was the last time I performed the query (updatedAt based polling).

From my understanding it uses the KVStore on the Heavy forwarder (I disabled it and saw the errors).

Which KVStore should the TA use? how does it work in a distributed environment?

Re: KV Store on Heavy Forwarder

woodcock — Fri, 27 Jul 2018 23:34:55 GMT

You are not even going to tell us the name of the TA or tell us where you got it?

Re: KV Store on Heavy Forwarder

rajrsplunk — Sat, 28 Jul 2018 14:39:35 GMT

Hey! it's a TA I'm currently developing and testing on Splunk distributed architecture.

Re: KV Store on Heavy Forwarder

DalJeanis — Sun, 29 Jul 2018 03:28:38 GMT

Okay, so you are building a TA (that's important information, not just "I have a TA Installed").

You are extracting data from APIs (which APIs? - that's important information.)

You are using the KVStore on the HF to store the state of the TA. (Why does the TA have a state? - that's important information.)

You disabled the KVStore on the HF. Why?

It seems like you probably need to join the Slack channel, and chat with the experienced folks down in the #appdevs subchannel (there are dozens of them that hang out there) about your use case.

Once you've gotten the big picture nailed down, you can either take the answer you found, post it as a comment to this answer, and accept this answer -- or you can ask one of the old hands to come post a clear, concise new answer as an explanation of your best approach, so you can accept that one.

Re: KV Store on Heavy Forwarder

rajrsplunk — Sun, 29 Jul 2018 12:09:55 GMT

Thank you for your response, I've edited the question. Since I'm relatively new to Splunk some of the questions are indeed lacking in information.

As edited, I'm extracting data from my application REST APIs (JSON format), I store the state to keep some data available for lookups, and also for remembering last checkpoint.

I disabled it as part of my experiments trying to understand how the KV works in a distributed environment.

I'll try the slack channel as well..