topic Re: How to write a Scripted input directly to lookup in Getting Data In

How to write a Scripted input directly to lookup?

amat — Fri, 25 Mar 2022 13:47:50 GMT

I have a requirement where I need to make an API call and write the data to a lookup file that I can use locally. The API calls returns data in a CSV format.

Previously, I used the Ad-on builder to create a python script that would make make the API request and index this data. However, I have a new requirement to skip the index entirely and write to a local lookup on the search head. The Ad-on builder wont help as it only shows examples of how to write the data to an index.

Thank you!

Re: Scripted input directly to lookup

PickleRick — Thu, 24 Mar 2022 20:21:45 GMT

The input itself (at least with the input functionality) cannot write to a lookup.

You need a script that manipulates lookup using REST API. It has nothing to do with indexing.

Re: How to write a Scripted input directly to lookup

richgalloway — Fri, 25 Mar 2022 00:15:01 GMT

You don't need the Add-on Builder to do that. You already have a Python script so just replace the part the indexes the data with a few lines of code to write it to ../lookups/mylookup.csv (or whatever you want to call the file).

Re: How to write a Scripted input directly to lookup

amat — Fri, 25 Mar 2022 02:55:55 GMT

interesting. So should I use the Ad-On builder but at the very end of the script have it overwrite the lookup table?

For background, the reason why I used the ad-on builder was because I was getting really confused with the authentication and safely pulling the API keys out of the passwords.conf. So the ad-on builder really helped with retrieval of secrets.

Are you suggesting overwriting the lookup using the REST endpoint? If so, how do i do that without authenticating ? I see a lot of the curl commands require you to pass admin credentials; however, i dont want to hardcode any creds in my script.

Re: How to write a Scripted input directly to lookup

richgalloway — Fri, 25 Mar 2022 12:17:28 GMT

AoB helps with the hard parts. Writing data to a disk file is not a hard part. Because scripted inputs run on the Splunk server, they have access to the file system there. Just use normal pythonic methods for opening and writing to a text file. You don't need REST.

Re: How to write a Scripted input directly to lookup?

VatsalJagani — Mon, 28 Mar 2022 06:33:13 GMT

I would suggest using a custom Python command with the help of Splunklib as input is not recommended on the search head. Here is some parts of the code:

commands.conf

[lookupgen] filename = lookup_gen.py chunked = true

lookup_gen.py

import os import sys import csv from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators HEADERS = ['ip','mac','hostname'] # change this as per your need LOOKUP_NAME = 'my_lookup.csv' @Configuration() class LookupGen(GeneratingCommand): def get_data_from_your_data_source(self): pass # write your logic to fetch the data here def update_lookup_file(self, lookup_file_path, data): with open(lookup_file_path, 'w') as f: csv_writer = csv.writer(f) csv_writer.writerow(HEADERS) csv_writer.writerows(data) def generate(self): data = self.get_data_from_your_data_source() lookup_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), os.path.join('lookups', LOOKUP_NAME)) self.update_lookup_file(lookup_path, data) dispatch(LookupGen, sys.argv, sys.stdin, sys.stdout, __name__)

You will need to add this python file into your bin folder along with splunklib (Python SDK for Splunk) - https://pypi.org/project/splunklib/

https://github.com/splunk/splunk-sdk-python

And you can schedule a search/report using this command at regular internal.