Getting Data In

How to write a Scripted input directly to lookup?

amat
Explorer

I have a requirement where I need to make an API call and write the data to a lookup file that I can use locally. The API calls returns data in a CSV format. 

 

Previously, I used the Ad-on builder to create a python script that would make make the API request and index this data. However, I have a new requirement to skip the index entirely and write to a local lookup on the search head. The Ad-on builder wont help as it only shows examples of how to write the data to an index.

 

Thank you!

Labels (3)
Tags (4)
0 Karma

VatsalJagani
Champion

I would suggest using a custom Python command with the help of Splunklib as input is not recommended on the search head. Here is some parts of the code:

 

commands.conf

 

[lookupgen]
filename = lookup_gen.py
chunked = true

 

 

lookup_gen.py

 

import os
import sys
import csv

from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators

HEADERS = ['ip','mac','hostname']   # change this as per your need
LOOKUP_NAME = 'my_lookup.csv'


@Configuration()
class LookupGen(GeneratingCommand):
    def get_data_from_your_data_source(self):
        pass # write your logic to fetch the data here

    def update_lookup_file(self, lookup_file_path, data):
        with open(lookup_file_path, 'w') as f:
            csv_writer = csv.writer(f)
            csv_writer.writerow(HEADERS)
            csv_writer.writerows(data)

 
    def generate(self):
        data = self.get_data_from_your_data_source()
        lookup_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
                                            os.path.join('lookups', LOOKUP_NAME))
        self.update_lookup_file(lookup_path, data)

 
dispatch(LookupGen, sys.argv, sys.stdin, sys.stdout, __name__)

 

 

You will need to add this python file into your bin folder along with splunklib (Python SDK for Splunk) - https://pypi.org/project/splunklib/

https://github.com/splunk/splunk-sdk-python

 

And you can schedule a search/report using this command at regular internal.

richgalloway
SplunkTrust
SplunkTrust

You don't need the Add-on Builder to do that.  You already have a Python script so just replace the part the indexes the data with a few lines of code to write it to ../lookups/mylookup.csv (or whatever you want to call the file).

---
If this reply helps you, an upvote would be appreciated.
0 Karma

amat
Explorer

interesting. So should I use the Ad-On builder but at the very end of the script have it overwrite the lookup table? 

For background, the reason why I used the ad-on builder was because I was getting really confused with the authentication and safely pulling the API keys out of the passwords.conf. So the ad-on builder really helped with retrieval of secrets. 

Are you suggesting overwriting the lookup using the REST endpoint? If so, how do i do that without authenticating ? I see a lot of the curl commands require you to pass admin credentials; however, i dont want to hardcode any creds in my script.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

AoB helps with the hard parts.  Writing data to a disk file is not a hard part.  Because scripted inputs run on the Splunk server, they have access to the file system there.  Just use normal pythonic methods for opening and writing to a text file.  You don't need REST.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

PickleRick
Ultra Champion

The input itself (at least with the input functionality) cannot write to a lookup.

You need a script that manipulates lookup using REST API. It has nothing to do with indexing.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...