How to write a Scripted input directly to lookup?

[amat]

I have a requirement where I need to make an API call and write the data to a lookup file that I can use locally. The API calls returns data in a CSV format. 

Previously, I used the Ad-on builder to create a python script that would make make the API request and index this data. However, I have a new requirement to skip the index entirely and write to a local lookup on the search head. The Ad-on builder wont help as it only shows examples of how to write the data to an index.

Thank you!

[VatsalJagani]

I would suggest using a custom Python command with the help of Splunklib as input is not recommended on the search head. Here is some parts of the code:

commands.conf

[lookupgen]
filename = lookup_gen.py
chunked = true

lookup_gen.py

import os
import sys
import csv

from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators

HEADERS = ['ip','mac','hostname']   # change this as per your need
LOOKUP_NAME = 'my_lookup.csv'


@Configuration()
class LookupGen(GeneratingCommand):
    def get_data_from_your_data_source(self):
        pass # write your logic to fetch the data here

    def update_lookup_file(self, lookup_file_path, data):
        with open(lookup_file_path, 'w') as f:
            csv_writer = csv.writer(f)
            csv_writer.writerow(HEADERS)
            csv_writer.writerows(data)

 
    def generate(self):
        data = self.get_data_from_your_data_source()
        lookup_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
                                            os.path.join('lookups', LOOKUP_NAME))
        self.update_lookup_file(lookup_path, data)

 
dispatch(LookupGen, sys.argv, sys.stdin, sys.stdout, __name__)

You will need to add this python file into your bin folder along with splunklib (Python SDK for Splunk) - https://pypi.org/project/splunklib/

https://github.com/splunk/splunk-sdk-python

And you can schedule a search/report using this command at regular internal.

[richgalloway]

You don't need the Add-on Builder to do that.  You already have a Python script so just replace the part the indexes the data with a few lines of code to write it to ../lookups/mylookup.csv (or whatever you want to call the file).

[amat]

interesting. So should I use the Ad-On builder but at the very end of the script have it overwrite the lookup table? 

For background, the reason why I used the ad-on builder was because I was getting really confused with the authentication and safely pulling the API keys out of the passwords.conf. So the ad-on builder really helped with retrieval of secrets. 

Are you suggesting overwriting the lookup using the REST endpoint? If so, how do i do that without authenticating ? I see a lot of the curl commands require you to pass admin credentials; however, i dont want to hardcode any creds in my script.

[richgalloway]

AoB helps with the hard parts.  Writing data to a disk file is not a hard part.  Because scripted inputs run on the Splunk server, they have access to the file system there.  Just use normal pythonic methods for opening and writing to a text file.  You don't need REST.

[PickleRick]

The input itself (at least with the input functionality) cannot write to a lookup.

You need a script that manipulates lookup using REST API. It has nothing to do with indexing.