All Apps and Add-ons
Highlighted

Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

Path Finder

Hey Gang,

Okay, let's start with the basics. We are running Splunk Enterprise 7.0.3 (on prem). We run in a full RHEL environment (I believe it's 7.0). We have an indexing cluster of 6 nodes spread across two sites. We have one machine that handles the duties of Master Node for the indexing cluster AND also takes care of the Deployer role for the search head cluster. We have a search head cluster of 4 nodes.

Now, here's the problem / question.

About a year ago (prior to being on a Search Head Cluster), I installed the Lookup File Editor so that my users could edit and maintain their own csv lookup files more easily. The users loved it. However, since moving to the SHC we've been noticing some weird side effects. The users will put in changes to a table. The table will work for several days, and then, their changes will "disappear". When this happens, there doesn't seem to be a "previous version" of the table that contains their changes for them to revert back to. We started noticing the problem with version 2.7.1 of the Lookup File Editor. As part of trouble shooting, I have upgraded to the latest version of the Lookup File Editor (3.0.4).

I have read some of the answers posts about changes not replicating properly, but that doesn't seem to be the behavior here. It almost seems like older versions of the tables are being replicated over the new changes, but I haven't been able to confirm that. Any thoughts or advice on how to troubleshoot (and hopefully fix) this would be very appreciated.

Thank you for your time,
Matt G.

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

Champion

This does sound somewhat like replication issues to me. The disappearing lookups could be due to users using different search heads and thus seeing whatever is on that search head. I am wondering if they could they be editing an older version of the lookup on a different search head which gets successfully replicated and thus overriding newer changes.

Have you confirmed that editing a lookup consistently causes a replication event to push it to the other search heads?

BTW: The lookup backups won't replicate unless you are on 3.0.x of the Lookup Editor and you enable the REST replay setting (allowRestReplay) in restmap.conf. I kept it off by default because there was a bug in 7.0.x which would cause Splunk to crash if you used this setting.

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

Path Finder

I haven't looked for a replication event after a change yet. Is there an easy way to do this (What am I saying? This is Splunk. OF COURSE there's an easy way to do it)? I'm not sure how to check for that.

If you can tell me what to look for, I will have the user execute a change on one of the lookups, and then I'll look for the replication event.

Thank you for the suggestion.

Based on what you said, I'm not sure I would want to make the change to restmap.conf being that we're still running 7.0.3, and I certainly wouldn't want Splunk to crash.

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

Champion

@mgranger1: I definitely wouldn't recommend setting the REST replay on 7.0.3. Even if you did, it likely would not be reliable since it sounds like replication may be having problems.

I'm not sure of all of troubleshooting steps for SHC but you may want to start here: http://docs.splunk.com/Documentation/Splunk/7.1.2/DistSearch/HowconfrepoworksinSHC#View_replication_...

If there are replication issues, then the Lookup Editor might not be the only thing affected too. Thus, it might be worth checking out in case issues crop up elsewhere.

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

SplunkTrust
SplunkTrust

any luck here? Sounds like replication to me as well, it's common with SHC.

But the question is old so I wanted to see if you ever found a solution and if so can we put that here and close this awarding points to LukeMurphy?

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

New Member

We are seeing similar issues here. Changes to our lookup tables keep reverting or disappearing. Were you able to find a resolution?

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

Path Finder

I found the answer to my question a while ago, but figured I would answer it now.

Something I wasn't aware of with the Search Head Cluster bundle is that it respects the default and local settings for everything EXCEPT lookup tables. Due to the fact that several applications rely on lookup tables as part of their configuration, when you push out a new Search Head Cluster bundle, you wind up pushing out all of the lookup tables that are on your Deployer. If, you have tables on your Search Head Cluster that have the same name as the ones that are on your Deployer, any changes that have been made to the copies on the Search Head Cluster will be overwritten by the original copies that are on your Deployer.

Therefore, you either need to periodically update the lookup tables on your Deployer by passing your "local" copies of the tables back up to the deployer, OR, you need to ensure that any local tables have different names than the ones on your deployer.

View solution in original post

0 Karma
Highlighted

Re: Changes disappearing from tables in Lookup File Editor since Search Head Cluster upgrade

SplunkTrust
SplunkTrust

There is some nuance here, the deployer overwrites the lookups on the search heads under certain circumstances (from memory it is when the modificaiton timestamp changes on the deployer but I have not tested this for a very long time)

However if you refer to Preserve lookup files across app upgrades in the docs then you can avoid the lookup files getting overwritten by the deployer...

0 Karma