All Apps and Add-ons

Lookup File Editor with Search Head Clustering? New Lookups not replicating

dhorn
Path Finder

So I've got an environment with 2 Search Head Pools, and one of the pools (with 4 search peers, and a dedicated deployer) has the Lookup File Editor app installed. If I use that app to create a new lookup, that new lookup is saved into the according app on that specific Search Head, but it's not being replicated to the other SH's in the pool.

I believe this is expected behavior after reading the Splunk docs on SH Clustering, but any idea if you can use your app to force replication of that new lookup? Knowledge Objects are supposed to be automatically replicated when created in the Splunk web UI, so I assume they are calling a specific action to case replication, or something along those lines.

Thought I'd ask here before opening a case with Splunk.

0 Karma

LukeMurphey
Champion

You are correct that Search Head Clustering (SHC) doesn't support this case. That leaves two options that I can think of:

Option 1: use KV store
Convert the lookup to use the KV store. KV store lookups are available on all search heads when using SHC. Unfortunately, I haven't added editing of KV store lookups to the lookup editor yet. I am planning on adding support for KV store files but I don't have an ETA yet.

Option 2: distribute lookups via another mechanism
You could find another mechanism for distributing the lookups and setting that up to keep the lookup file in sync. For example, I wrote an app that allows you to import lookup files from Google Docs. In that case, you could store the lookup file in Google and edit it using their tools and then have an input that downloads it to your search heads.

Update:
Version 2.0.0 of the lookup editor supports editing KV store lookups. You can download it from Splunk-base. Note that you will select version 2.0.0 from the dropdown for now because I'm looking to get some customers feedback before I set it as the active release. Please let me know if you use it and let me know if you see any issues (or if everything worked perfectly).

jkmurthy
Explorer

Hi Luke,

My understanding is that lookup files as a knowledge object do get replicated in a search head clustered environment. However, based on the above response, is it correct to state that lookup files modified via lookup editor do not get replicated in the cluster ?

We currently use lookups to store our application configurations, some of which could be modified by the end users. As you have mentioned, using a kvstore could be an option if we need replication, but the lookup editor app provides a very user friendly interface to modify values and hence achieving similar thing for kvstore would mean developing all of that.

Are there options that we could think of to continue to use lookups (and modify them via lookup editor and replicate them in the cluster) ? Here are few thoughts. Please let me know if it makes sense:
1. Once the lookups are modified, can they be pushed programatically to the deployer and invoke a script to replicate across all search heads ? If you have some idea about this, please do share.
2. As per the documentation REST API calls are replicated, so is it possible that once the lookup is modified, we trigger a REST API call to do something like :
|inputlookup | outputlookup . Would this then be actually considered for replication ?
3. Would it work if we enable lookup editor app in deployer itself and modify the changes there and push it to other components ? am not sure if the deployer can provide a web interface for this. Please do confirm.

Regards
Keshav

0 Karma

LukeMurphey
Champion

What I wrote may not be true anymore because Splunk has been adding additional features to SHC such that it replicates more. I don't have a system setup with SHC to verify though.

I am working on an update to the lookup editor that would allow it to modify KV store lookups. Once done, that should solve this problem. I'm hoping to have this done by the end of the month.

My understanding is that modifying the lookup on the conf deployer would work though I haven't tested it. That might be the best intermediate option. In the meantime, I'll try to get the app updated with KV support soon.

0 Karma

ashleyherbert
Communicator

Hi Luke,
Just wondering if you're still looking to get this working with SH Clustering? We use the lookup editor quite heavily to manage recipients of alerts (combined with the sendresults command). The app is really handy, but we've just changed our test environment to use SH clustering and found we can no longer use this.
Just want to find out if we should hold fire on redesigning our solution, in case you have an update planned?
Thanks,
Ashley

0 Karma

LukeMurphey
Champion

@ashleyherbert: I have a version 2.0 that adds SHC compatibility available (you just need to select version 2.0 from the dropdown on the Lookup Editor Splunkbase page). I'm finalizing a couple more things before I make it the default version. I was also hoping to get some feedback from users that this version is working for them.

0 Karma

LukeMurphey
Champion

I just pushed a new version of 2.0.0. I haven't made it the default version yet but this not only allows you to use and replicate KV store lookups but it will also allow CSV lookups to be replicated. The CSV lookups won't likely replicate until you save it. Thus, it won't show up everywhere in your environment until you load the file and press save.

0 Karma

LukeMurphey
Champion

BTW: please let me know if you test it. I would like to get feedback before I make 2.0.0 the default. It works perfectly for me, but I would love to have feedback from someone else.

0 Karma

rajkumargopagon
Explorer

Hi Luke,
I installed the new version 2.0.0 in our Search Head Cluster. When trying to create a new lookup file, in the apps list, it shows only first 30 apps(sorted alphabetically). Is there a limit on the number of apps being shown in the list.

0 Karma

LukeMurphey
Champion

That list should only include apps that have lookup files (I filter apps without lookups). Are you finding that it is not showing apps that do have lookup files?

0 Karma

rajkumargopagon
Explorer

Correct it doesn't list all apps with lookups. It even shows apps which do not have any lookups as well. It basically shows the first 30 apps alphabetically. Shouldn't it show all the apps in the list so that whenever I try to create a new lookup, I can choose to add the lookup in any app.

0 Karma

LukeMurphey
Champion

It should only be filtering them on the main list page, not the page for creating a new lookup. The new lookup page should be listing all of them. Let me look into that.

0 Karma

SwatiApte
Path Finder

Hi Luke,

I have downloaded and installed the latest version (2.0.2) of the Lookup Editor App. I found two issues with it that I want to inform about:

  1. The Apps list is not available in the "App" Filter on the "Lookups" Dashboard. Also, none of the Lookups created in different Apps, are listed on this Dashboard. The permissions for the Lookups is Global, so ideally they should have been listed here. The KVStore created, is listed however.

  2. While Editing a KVStore, there is no "Save" button available to save the changes.

Could you please let me know if they are valid issues, or is there some problem with my Splunk instance itself, like some setting that I might be missing?

0 Karma

LukeMurphey
Champion

Yep, that is definitely broken. I'll fix it ASAP.

0 Karma

LukeMurphey
Champion

@rajkumargopagoni: I got a fix for that issue. I updated version 2.0.0 accordingly. Thanks for the feedback, I really appreciate it. You might have to restart Splunk and/or clear the browser cache to get the changes to be recognized by the browser.

0 Karma

ashleyherbert
Communicator

Hi Luke, thanks very much, I didn't notice that there was another version there. We're downloading it now and will test it over the next couple of days. I'll let you know if we find any problems.

Cheers,
Ash

0 Karma

jkmurthy
Explorer

Thanks Luke, unfortunately even we do not yet have a test environment which is cluster enabled. We will try if we can get it tested in any way. Would look forward for the updated app as well.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!