I have a Search Head cluster setup. Within the search app, I have defined a number of lookups, which I would like to update regularly.
The lookup table comes from a number of sources (e.g. information from spreadsheet converted to CSV with multiple tabs within the spreadsheet, db dump into csv).
The CSV can be created on a regular basis and I need to update the CSVs on the search head members.
The lookups seem to be synchronized when created via the web interface. However, if I update the files manually, it does not replicate across.
What is the best way to update lookup tables manually (planning to automate this via cronjob scripts)?
Hi,
you need to do the changes on the deployer, than it will bei sync to the other members.
The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files.
For the default search app to replicate to the cluster members, you can achieve this via Rest API.
By updating one of the Search head, it will then replicate to the rest of the Search Head cluster members.
This is the Splunk expected behaviour, in order to replicate the updated lookups in distributed environment, you’ll need to use the deployer to push it to all cluster members.
As noted in the Splunk documentation:
"For example, if a user creates a saved search in Splunk Web on a cluster member, the cluster replicates that saved search to all cluster members. However, if you, as the administrator, add a saved search by directly editing the savedsearches.conf file on one cluster member, the cluster does not replicate that saved search to the other cluster members. You must use the deployer to push that saved search to all cluster members."
Please refer to the following URL under the header “Configuration methods that trigger replication”:
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/HowconfrepoworksinSHC
Here are the steps to update the lookup table. To automate this, it can be achieved by putting these steps to script.
Example:
I use the geo_attr_us_states.csv lookup file for update.
Setting up the new file in the staging directory, it is in the lookup_tmp:
/opt/splunk/var/run/splunk/lookup_tmp/geo_attr_us_states.csv
Currently, the file geo_attr_us_states.csv already exist in the lookup directory:
/opt/splunk/etc/apps/search/lookups
Since this file is going to be updated, I move this file to temporary directory. Otherwise, it will give error that object is already exist.
mv /opt/splunk/etc/apps/search/lookups/geo_attr_us_states.csv /tmp
Using REST API, update the lookup file from the staging directory to Post to Lookup table:
$ curl -k -u admin https://localhost:8089/servicesNS/nobody/search/data/lookup-table-files -d eai:data=/opt/splunk/var/run/splunk/lookup_tmp/geo_attr_us_states.csv -d name=geo_attr_us_states.csv
You will find that the file is Posted to the lookup directory:
ls /opt/splunk/etc/apps/search/lookups
After the lookup file is updated, the file in the staging directory is removed.
ls /opt/splunk/var/run/splunk/lookup_tmp
Check the lookup file in the other cluster members.
I verified that the lookup is replicating to the cluster members in my lab.
ls /opt/splunk/etc/apps/search/lookups/geo_attr_us_states.csv
The replicated file will show with the modified date and time.
Repeat step 1-7 again to update with the file.
Replicates to other search heads automatically
Hi,
you need to do the changes on the deployer, than it will bei sync to the other members.
The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files.