All Apps and Add-ons

Rest API Modular Input: Combining multiple REST URLs?

david_rose
Communicator

I am trying to pull volume data in from my VMAX array. I have about 200 volumes and I need a unique REST call for each one.

The call looks like this:

https://unisphere:8443/univmax/restapi/provisioning/symmetrix/XXXXXXXXXX/volume/<VOLUMEID>;

Is there a way to programmatically populate the <VOLUMEID> while iterating call through a list? I know you can do $foo$ substitution, but it seems to me like that is for single instances, not multiple.

Thanks!

1 Solution

Damien_Dallimor
Ultra Champion

The latest release of the REST API Modular Input v1.3.7 , now supports tokens that can return a list of values.

Token replacement functions in the URL can return a list of values that will cause multiple URL's to be formed and the requests for these URL's will be executed in parallel in multiple threads.

See tokens.py for an example.

So for your use case above just create a function in tokens.py that returns a list of volume id's

def volumeids():
    return ['123','456','789']

And then setup your URL like :

https://unisphere:8443/univmax/restapi/provisioning/symmetrix/XXXXXXXXXX/volume/$volumeids$

View solution in original post

Damien_Dallimor
Ultra Champion

The latest release of the REST API Modular Input v1.3.7 , now supports tokens that can return a list of values.

Token replacement functions in the URL can return a list of values that will cause multiple URL's to be formed and the requests for these URL's will be executed in parallel in multiple threads.

See tokens.py for an example.

So for your use case above just create a function in tokens.py that returns a list of volume id's

def volumeids():
    return ['123','456','789']

And then setup your URL like :

https://unisphere:8443/univmax/restapi/provisioning/symmetrix/XXXXXXXXXX/volume/$volumeids$

DanielFordWA
Contributor

This is awesome, just came across this problem and its already solved!

david_rose
Communicator

Aaaaaaaand this is why the splunk community is the best community. Thanks Damien!

Damien_Dallimor
Ultra Champion

I aim to please 🙂

0 Karma

Damien_Dallimor
Ultra Champion

As you correctly note , you can do a substitution on a single URL , but with the current code this would still require you to setup 200 different REST input stanzas.

Your best bet might be to fork a copy of the code on Github and create your own "multiplexed" hack for your specific needs.
Should only require very minimal code.

Perhaps you could add some code around the "replaceTokens" method in rest.py to check if the token substitution function from tokens.py returns 1 or more results , and if it returns multiple results , then perform multiple URL requests in a loop rather than just 1 single request.

0 Karma

Damien_Dallimor
Ultra Champion

Update : I got the itch to code and I am working on this for you.Hang tight.

david_rose
Communicator

Thanks for that! I think it will open up a ton of flexibility for the input.

0 Karma

phil_iw
New Member

hi damien, does that mean you'll be able to make multiple substitutions on a single url? would that enable a kind of cartesian product?

0 Karma

Damien_Dallimor
Ultra Champion

Correct.Multiple requests will get spawned in parallel or optionally sequentially.If you have multiple tokens in your URL that return lists , then "Cartesianing" will occur. But all responses to these requests will still get indexed as individual events in Splunk.

Trivial example :

Lets say you create these 2 token functions :

def sometoken():
    return ['goo','foo']

def anothertoken():
    return ['boo','hoo']

Will spawn 2 requests :

http://foo.com?param1=$sometoken$

Will spawn 4 requests :

http://foo.com?param1=$sometoken$&param2=$anothertoken$
0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...