Building for the Splunk Platform

Is this Dynamic Data Input possible?

jlunde
Loves-to-Learn

This might be impossible, but thought I would at least ask the question before giving up! I have created an add-on that uses python scripts to pull in and index/checkpoint data. I am already successfully pulling from several of these data input sources. The issue I am running into is that the next API I want to pull data from requires an identifier at the end (a UUID in this case).  I dont know this uuid until I do a search on one of the forementioned inputs. 

So my question is, is there a way to take a value (variable/token/etc...) and dynamically create another data input from it? Can you define a $var$ in your script that can be dynamically passed in, in order to index data from that endpoint?

Labels (3)
0 Karma

PickleRick
Ultra Champion

I'm not sure I follow...

You're making a scripted or modular input, right?

You're calling some external API to fetch the data from there?

And you want to be able to search already indexed data in splunk for some parameter that you would use to call that API?

Technically it'a possible - after all a modular input is just another process which outputs data which Splunk ingests. If it does many different things under the hood noone cares. So theoretically you could call Splunk back using API and spawn search, read its results and use it.

But.

It's gonna be helluva difficult to write something like that right. Error handling, managing network connectivity (unless you're planning to only ever run it on an all-in-one) and so on.

Often the modular inputs simply keep internal state (if needed) in some local db.

0 Karma

jlunde
Loves-to-Learn

I am fairly new to splunking (if we are being honest), but thank you for your response. Below is some cleaned-up code of what I am trying to do. The issue is that I dont know the 'uuid' variable until I run some queries on other data sources.

 

def collect_events(helper, ew):
    opt_api_key=str(helper.get_global_setting('csw_api_key'))
    opt_api_secret=str(helper.get_global_setting('csw_api_secret'))
    API_ENDPOINT=str(helper.get_global_setting('csw_url'))
    opt_uuid=str(helper.get_arg('uuid'))
    #auth
    restclient = RestClient(API_ENDPOINT, api_key=opt_api_key, api_secret=opt_api_secret, verify=False)
    #GET
    resp = restclient.get('/workload/'+ opt_uuid)
    #Turn Resp into python list
    r_status=resp.status_code
    if r_status !=200:
        resp.raise_for_status()
    parsed_resp = resp.json()
    event = helper.new_event(json.dumps(parsed_resp), time=None, host=None, index=None, source=None, sourcetype=None, done=True, unbroken=True)
    ew.write_event(event)

 

0 Karma

PickleRick
Ultra Champion

Ok. But what do you mean by "other data sources"? Other inputs from which you get data into splunk? And those parameters are getting ingested into splunk as events? Or do you simply have to call another external API or something like that?

0 Karma

jlunde
Loves-to-Learn

Sorry PickleRick, have been heads down on some stuff the last few days. So to answer your question, yes, I can get the uuid I need from an existing event inside splunk. 

I, however, had one of those famous IT 'aha' moments after I stepped away from this for a bit. I retrofitted my add-on python code to just glean the uuid's through its own API call, put them in a list, and then loop through them as part of what it does to get the data. So all-in-all, problem solved. Thanks for your assistance!

0 Karma

PickleRick
Ultra Champion

👍

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...