Splunk Search

How do you update a lookup table manually in a distributed search head environment?

rsantoso_splunk
Splunk Employee
Splunk Employee

I have a Search Head cluster setup. Within the search app, I have defined a number of lookups, which I would like to update regularly.
The lookup table comes from a number of sources (e.g. information from spreadsheet converted to CSV with multiple tabs within the spreadsheet, db dump into csv).

The CSV can be created on a regular basis and I need to update the CSVs on the search head members.

The lookups seem to be synchronized when created via the web interface. However, if I update the files manually, it does not replicate across.

What is the best way to update lookup tables manually (planning to automate this via cronjob scripts)?

0 Karma
1 Solution

dkeck
Influencer

Hi,

you need to do the changes on the deployer, than it will bei sync to the other members.

The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files. 

http://docs.splunk.com/Documentation/Splunk/7.2.1/DistSearch/HowconfrepoworksinSHC#Configuration_met...

View solution in original post

0 Karma

rsantoso_splunk
Splunk Employee
Splunk Employee

For the default search app to replicate to the cluster members, you can achieve this via Rest API.
By updating one of the Search head, it will then replicate to the rest of the Search Head cluster members.

This is the Splunk expected behaviour, in order to replicate the updated lookups in distributed environment, you’ll need to use the deployer to push it to all cluster members.

As noted in the Splunk documentation:
"For example, if a user creates a saved search in Splunk Web on a cluster member, the cluster replicates that saved search to all cluster members. However, if you, as the administrator, add a saved search by directly editing the savedsearches.conf file on one cluster member, the cluster does not replicate that saved search to the other cluster members. You must use the deployer to push that saved search to all cluster members."
Please refer to the following URL under the header “Configuration methods that trigger replication”:
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/HowconfrepoworksinSHC

Here are the steps to update the lookup table. To automate this, it can be achieved by putting these steps to script.

Example:
I use the geo_attr_us_states.csv lookup file for update.

  1. Setting up the new file in the staging directory, it is in the lookup_tmp:
    /opt/splunk/var/run/splunk/lookup_tmp/geo_attr_us_states.csv

  2. Currently, the file geo_attr_us_states.csv already exist in the lookup directory:
    /opt/splunk/etc/apps/search/lookups

  3. Since this file is going to be updated, I move this file to temporary directory. Otherwise, it will give error that object is already exist.
    mv /opt/splunk/etc/apps/search/lookups/geo_attr_us_states.csv /tmp

  4. Using REST API, update the lookup file from the staging directory to Post to Lookup table:
    $ curl -k -u admin https://localhost:8089/servicesNS/nobody/search/data/lookup-table-files -d eai:data=/opt/splunk/var/run/splunk/lookup_tmp/geo_attr_us_states.csv -d name=geo_attr_us_states.csv

  5. You will find that the file is Posted to the lookup directory:
    ls /opt/splunk/etc/apps/search/lookups

  6. After the lookup file is updated, the file in the staging directory is removed.
    ls /opt/splunk/var/run/splunk/lookup_tmp

  7. Check the lookup file in the other cluster members.
    I verified that the lookup is replicating to the cluster members in my lab.
    ls /opt/splunk/etc/apps/search/lookups/geo_attr_us_states.csv
    The replicated file will show with the modified date and time.

Repeat step 1-7 again to update with the file.

Replicates to other search heads automatically

0 Karma

dkeck
Influencer

Hi,

you need to do the changes on the deployer, than it will bei sync to the other members.

The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files. 

http://docs.splunk.com/Documentation/Splunk/7.2.1/DistSearch/HowconfrepoworksinSHC#Configuration_met...

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...