Is this Dynamic Data Input possible?

[jlunde]

This might be impossible, but thought I would at least ask the question before giving up! I have created an add-on that uses python scripts to pull in and index/checkpoint data. I am already successfully pulling from several of these data input sources. The issue I am running into is that the next API I want to pull data from requires an identifier at the end (a UUID in this case).  I dont know this uuid until I do a search on one of the forementioned inputs. 

So my question is, is there a way to take a value (variable/token/etc...) and dynamically create another data input from it? Can you define a $var$ in your script that can be dynamically passed in, in order to index data from that endpoint?

[PickleRick]

I'm not sure I follow...

You're making a scripted or modular input, right?

You're calling some external API to fetch the data from there?

And you want to be able to search already indexed data in splunk for some parameter that you would use to call that API?

Technically it'a possible - after all a modular input is just another process which outputs data which Splunk ingests. If it does many different things under the hood noone cares. So theoretically you could call Splunk back using API and spawn search, read its results and use it.

But.

It's gonna be helluva difficult to write something like that right. Error handling, managing network connectivity (unless you're planning to only ever run it on an all-in-one) and so on.

Often the modular inputs simply keep internal state (if needed) in some local db.

[jlunde]

I am fairly new to splunking (if we are being honest), but thank you for your response. Below is some cleaned-up code of what I am trying to do. The issue is that I dont know the 'uuid' variable until I run some queries on other data sources.

def collect_events(helper, ew):
    opt_api_key=str(helper.get_global_setting('csw_api_key'))
    opt_api_secret=str(helper.get_global_setting('csw_api_secret'))
    API_ENDPOINT=str(helper.get_global_setting('csw_url'))
    opt_uuid=str(helper.get_arg('uuid'))
    #auth
    restclient = RestClient(API_ENDPOINT, api_key=opt_api_key, api_secret=opt_api_secret, verify=False)
    #GET
    resp = restclient.get('/workload/'+ opt_uuid)
    #Turn Resp into python list
    r_status=resp.status_code
    if r_status !=200:
        resp.raise_for_status()
    parsed_resp = resp.json()
    event = helper.new_event(json.dumps(parsed_resp), time=None, host=None, index=None, source=None, sourcetype=None, done=True, unbroken=True)
    ew.write_event(event)

[PickleRick]

Ok. But what do you mean by "other data sources"? Other inputs from which you get data into splunk? And those parameters are getting ingested into splunk as events? Or do you simply have to call another external API or something like that?

[jlunde]

Sorry PickleRick, have been heads down on some stuff the last few days. So to answer your question, yes, I can get the uuid I need from an existing event inside splunk. 

I, however, had one of those famous IT 'aha' moments after I stepped away from this for a bit. I retrofitted my add-on python code to just glean the uuid's through its own API call, put them in a list, and then loop through them as part of what it does to get the data. So all-in-all, problem solved. Thanks for your assistance!

[PickleRick]

👍