All Apps and Add-ons

REST API Modular Input - Tokens in URL (Exit Status 1)

efullman
Path Finder

Hi,

I'm trying to use tokens in the URL Arguments. (I'm hoping I can put the tokens there rather than just the endpoint. I actually tried cutting and pasting my URL arguments with my tokens into the actual URL Endpoint field with a ?, and it still got the same exception. I'm thinking you can put these tokens in the URL Arguments field.)

Anyway, I'm trying to do a special SQL call to the REST endpoint that includes "$" in the SQL that aren't part of the tokens I'm using. I'm thinking this may be a problem, and they may need to be escaped or something to avoid causing a conflict with the exception handler.

The REST API setup worked before to pull the bulk of the data in the dataset through this endpoint. So I'm not worried about the endpoint or anything, and I'm not getting far enough now for an error from the REST endpoint itself.

I'm just now customizing the configuration so that it makes a call using this special SQL with a cron in the "Polling Interval" to keep the dataset up to date.

I'm on version 1.3.9.

Here is what my URL arguments look like:

$order=reported_on DESC^$where=reported_on between $starttime_mten$ and $timenow$^$limit=50000

Here is the code I added to tokens.py:

def starttime_mten():
    tenmins = datetime.timedelta(minutes=10)
    now = datetime.data.now()
    timemten = now - tenmins
    return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow()
    now = datetime.data.now()
    return timenow.strftime('%Y-%m-%dT%H:%M:%S')

Here is the error log:

07-18-2015 21:39:12.981 +0000 WARN  ModularInputs - Validation for scheme=rest failed: The script returned with exit status 1.
07-18-2015 21:39:12.981 +0000 INFO  ModularInputs - The script returned with exit status 1.

I also restarted just to make sure there wasn't a problem otherwise, and it failed to load again. Here's that error from the log:

07-18-2015 22:00:54.157 +0000 ERROR ModularInputs - Introspecting scheme=rest: script running failed (exited with code 1).
07-18-2015 22:00:54.157 +0000 ERROR ModularInputs - Unable to initialize modular input "rest"  defined inside the app "rest_ta": Introspecting scheme=rest: script running failed (exited with code 1).

Additionally I have a delimeter set of "^"

And the "Polling Interval" of:

*/10 * * * * *

Any thoughts on why I'm getting this error.

Thanks in advance for the help.

Tags (1)
1 Solution

efullman
Path Finder

Answered my own question.

Turns out this exit status was because I had serious errors in my python code.

The final python that at least passed exception handling (we'll see if it works in the SQL), was:

def starttime_mten():
     tenmins = datetime.timedelta(minutes=10)
     now = datetime.date.now()
     timemten = now - tenmins
     return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
     now = datetime.date.now()
     return now.strftime('%Y-%m-%dT%H:%M:%S')

In order to fix the load error that caused the rest_ta not to load, I had to delete my Python changes. Then it loaded okay. Then I went back in and essentially debugged my buggy python.

View solution in original post

Damien_Dallimor
Ultra Champion

If you want to token replace url arguments , then encode them into the endpoint url parameter rather than declaring them in the url arguments parameter

alt text

upmangaurav
Explorer

Hi Damien/efullman, Can you guys please drop a hint about where to declare these variable tokens specified inside Endpoint URL. I have a few tokens inserted into the endpoint URL which I want to be fetched from some python class. But I can't seem to figure out where to put/insert that python code in the /rest_ta/bin directory?

0 Karma

Damien_Dallimor
Ultra Champion

Have a look at the token substitution section in the documentation.

rest_ta/bin/tokens.py

0 Karma

upmangaurav
Explorer

Thanks Damien, That did it!

0 Karma

efullman
Path Finder

Thanks that worked as expected. I think adding token replacement in the other terms besides the endpoint would be a good improvement for the future.

0 Karma

efullman
Path Finder

Thanks I'll give it a try

0 Karma

efullman
Path Finder

I just noticed your comment that the tokens don't work in the URL arguments.

I was looking through the rest.py code. It seems like we could call the replaceTokens function for the url_args. As you know this "else" is near the end of the code.

I tried to hack a mod in here, but my python skills are weak. This didn't work, as a second "do_run", how would you suggest I could hack this in?

    else:
    config = get_input_config()
        original_endpoint=config.get("endpoint")
        #token replacement
        endpoint_list = replaceTokens(original_endpoint)
        #token replacement url args
        original_url_args=config.get("url_args")
        url_arg_list = replaceTokens(original_url_args)

        sequential_mode=int(config.get("sequential_mode",0))

        if bool(sequential_mode):
            do_run(config,endpoint_list)
         #token for url args
            do_run(config,url_arg_list)
        else:  #parallel mode
            for endpoint in endpoint_list:
                requester = threading.Thread(target=do_run, args=(config,[endpo$
                requester.start()
0 Karma

efullman
Path Finder

Thanks for your input, I did see the error in the cron. I've since realized I can't poll the database this frequently I need to build a more extensive parser that saves the data and time of the last event, and then starts the search with that value.

Before I do that I need to implement something brute force that will execute with the cron: 0 0 * * * (e.g. every midnight).

My new python is simpler, just figuring out yesterday's date.

def start_day():
     one_day = datetime.timedelta(days=1)
     today = datetime.date.today()
     startday = today - one_day
     return startday.strftime('%Y-%m-%d')

I'm actually doing a call with URL arguments that include:

$where=applieddate between $start_day$ and $datetoday$

However, I'm getting an error, and I think its because the tokens aren't replacing. I'm getting a 400, but the return as error is logged into the index and it reads:

http_error_code = 400 error_message = {"message":"query.soql.no-such-column","errorCode":"query.soql.no-such-column","data":{"data":{"column":"$start_day$","dataset":"alpha.34796","position":{"row":1,"column":38,"line":"SELECT * WHERE `applieddate` BETWEEN `$start_day$` AND `$datetoday$` ORDER BY `applieddate` DESC NULL FIRST\n      

I would have expected the two tokens to be replaced with the date, since this is coming back from the endpoint into the index. Any ideas why the python wouldn't have executed?

I saved it into the tokens.py file, and then I restarted Splunk for good measure in case it needed to be loaded in an include or something.

Your ideas are appreciated.

0 Karma

Damien_Dallimor
Ultra Champion

Your cron pattern has 1 too many "*".

*/10 * * * * not */10 * * * * *

Token replacement only applys to the endpoint field , not the url_args field.

efullman
Path Finder

Answered my own question.

Turns out this exit status was because I had serious errors in my python code.

The final python that at least passed exception handling (we'll see if it works in the SQL), was:

def starttime_mten():
     tenmins = datetime.timedelta(minutes=10)
     now = datetime.date.now()
     timemten = now - tenmins
     return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
     now = datetime.date.now()
     return now.strftime('%Y-%m-%dT%H:%M:%S')

In order to fix the load error that caused the rest_ta not to load, I had to delete my Python changes. Then it loaded okay. Then I went back in and essentially debugged my buggy python.

efullman
Path Finder

More refinement. Needed datatime not date.

def starttime_mten():
     tenmins = datetime.timedelta(minutes=10)
     now = datetime.datetime.now()
     timemten = now - tenmins
     return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
     now = datetime.datetime.now()
     return now.strftime('%Y-%m-%dT%H:%M:%S')
0 Karma

efullman
Path Finder

Ignore the "5." and "10." in the tokens.py code. That was injected by the editor when I pasted it.

0 Karma

efullman
Path Finder

One more thing. I cancelled out of the edit of the REST API, and it already had the configuration that didn't work in it.

When I restarted it got the introspection error.

Now the app doesn't seem to be loaded.

The Data Input window doesn't come up with REST as one of the choices. I'll need some help to identify where to edit that configuration file so that the REST can load. I assume I'll have to fix that first and restart before we can fix this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...