I have a requirement where I need to make an API call and write the data to a lookup file that I can use locally. The API calls returns data in a CSV format.
Previously, I used the Ad-on builder to create a python script that would make make the API request and index this data. However, I have a new requirement to skip the index entirely and write to a local lookup on the search head. The Ad-on builder wont help as it only shows examples of how to write the data to an index.
I would suggest using a custom Python command with the help of Splunklib as input is not recommended on the search head. Here is some parts of the code:
[lookupgen] filename = lookup_gen.py chunked = true
import os import sys import csv from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators HEADERS = ['ip','mac','hostname'] # change this as per your need LOOKUP_NAME = 'my_lookup.csv' @Configuration() class LookupGen(GeneratingCommand): def get_data_from_your_data_source(self): pass # write your logic to fetch the data here def update_lookup_file(self, lookup_file_path, data): with open(lookup_file_path, 'w') as f: csv_writer = csv.writer(f) csv_writer.writerow(HEADERS) csv_writer.writerows(data) def generate(self): data = self.get_data_from_your_data_source() lookup_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), os.path.join('lookups', LOOKUP_NAME)) self.update_lookup_file(lookup_path, data) dispatch(LookupGen, sys.argv, sys.stdin, sys.stdout, __name__)
You will need to add this python file into your bin folder along with splunklib (Python SDK for Splunk) - https://pypi.org/project/splunklib/
And you can schedule a search/report using this command at regular internal.
You don't need the Add-on Builder to do that. You already have a Python script so just replace the part the indexes the data with a few lines of code to write it to ../lookups/mylookup.csv (or whatever you want to call the file).
interesting. So should I use the Ad-On builder but at the very end of the script have it overwrite the lookup table?
For background, the reason why I used the ad-on builder was because I was getting really confused with the authentication and safely pulling the API keys out of the passwords.conf. So the ad-on builder really helped with retrieval of secrets.
Are you suggesting overwriting the lookup using the REST endpoint? If so, how do i do that without authenticating ? I see a lot of the curl commands require you to pass admin credentials; however, i dont want to hardcode any creds in my script.
AoB helps with the hard parts. Writing data to a disk file is not a hard part. Because scripted inputs run on the Splunk server, they have access to the file system there. Just use normal pythonic methods for opening and writing to a text file. You don't need REST.