topic Re: REST API Modular Input - Tokens in URL (Exit Status 1) in All Apps and Add-ons

REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Sat, 18 Jul 2015 22:09:18 GMT

Hi,

I'm trying to use tokens in the URL Arguments. (I'm hoping I can put the tokens there rather than just the endpoint. I actually tried cutting and pasting my URL arguments with my tokens into the actual URL Endpoint field with a ?, and it still got the same exception. I'm thinking you can put these tokens in the URL Arguments field.)

Anyway, I'm trying to do a special SQL call to the REST endpoint that includes "$" in the SQL that aren't part of the tokens I'm using. I'm thinking this may be a problem, and they may need to be escaped or something to avoid causing a conflict with the exception handler.

The REST API setup worked before to pull the bulk of the data in the dataset through this endpoint. So I'm not worried about the endpoint or anything, and I'm not getting far enough now for an error from the REST endpoint itself.

I'm just now customizing the configuration so that it makes a call using this special SQL with a cron in the "Polling Interval" to keep the dataset up to date.

I'm on version 1.3.9.

Here is what my URL arguments look like:

$order=reported_on DESC^$where=reported_on between $starttime_mten$ and $timenow$^$limit=50000

Here is the code I added to tokens.py:

def starttime_mten():
    tenmins = datetime.timedelta(minutes=10)
    now = datetime.data.now()
    timemten = now - tenmins
    return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow()
    now = datetime.data.now()
    return timenow.strftime('%Y-%m-%dT%H:%M:%S')

Here is the error log:

07-18-2015 21:39:12.981 +0000 WARN  ModularInputs - Validation for scheme=rest failed: The script returned with exit status 1.
07-18-2015 21:39:12.981 +0000 INFO  ModularInputs - The script returned with exit status 1.

I also restarted just to make sure there wasn't a problem otherwise, and it failed to load again. Here's that error from the log:

07-18-2015 22:00:54.157 +0000 ERROR ModularInputs - Introspecting scheme=rest: script running failed (exited with code 1).
07-18-2015 22:00:54.157 +0000 ERROR ModularInputs - Unable to initialize modular input "rest"  defined inside the app "rest_ta": Introspecting scheme=rest: script running failed (exited with code 1).

Additionally I have a delimeter set of "^"

And the "Polling Interval" of:

*/10 * * * * *

Any thoughts on why I'm getting this error.

Thanks in advance for the help.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Sat, 18 Jul 2015 22:20:09 GMT

Ignore the "5." and "10." in the tokens.py code. That was injected by the editor when I pasted it.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Sat, 18 Jul 2015 22:28:07 GMT

One more thing. I cancelled out of the edit of the REST API, and it already had the configuration that didn't work in it.

When I restarted it got the introspection error.

Now the app doesn't seem to be loaded.

The Data Input window doesn't come up with REST as one of the choices. I'll need some help to identify where to edit that configuration file so that the REST can load. I assume I'll have to fix that first and restart before we can fix this.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Sat, 18 Jul 2015 23:10:48 GMT

Answered my own question.

Turns out this exit status was because I had serious errors in my python code.

The final python that at least passed exception handling (we'll see if it works in the SQL), was:

def starttime_mten():
     tenmins = datetime.timedelta(minutes=10)
     now = datetime.date.now()
     timemten = now - tenmins
     return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
     now = datetime.date.now()
     return now.strftime('%Y-%m-%dT%H:%M:%S')

In order to fix the load error that caused the rest_ta not to load, I had to delete my Python changes. Then it loaded okay. Then I went back in and essentially debugged my buggy python.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Sun, 19 Jul 2015 00:44:25 GMT

More refinement. Needed datatime not date.

def starttime_mten():
     tenmins = datetime.timedelta(minutes=10)
     now = datetime.datetime.now()
     timemten = now - tenmins
     return timemten.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
     now = datetime.datetime.now()
     return now.strftime('%Y-%m-%dT%H:%M:%S')

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

Damien_Dallimor — Sun, 19 Jul 2015 02:46:32 GMT

Your cron pattern has 1 too many "*".

*/10 * * * * not */10 * * * * *

Token replacement only applys to the endpoint field , not the url_args field.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Mon, 20 Jul 2015 05:11:35 GMT

Thanks for your input, I did see the error in the cron. I've since realized I can't poll the database this frequently I need to build a more extensive parser that saves the data and time of the last event, and then starts the search with that value.

Before I do that I need to implement something brute force that will execute with the cron: 0 0 * * * (e.g. every midnight).

My new python is simpler, just figuring out yesterday's date.

def start_day():
     one_day = datetime.timedelta(days=1)
     today = datetime.date.today()
     startday = today - one_day
     return startday.strftime('%Y-%m-%d')

I'm actually doing a call with URL arguments that include:

$where=applieddate between $start_day$ and $datetoday$

However, I'm getting an error, and I think its because the tokens aren't replacing. I'm getting a 400, but the return as error is logged into the index and it reads:

http_error_code = 400 error_message = {"message":"query.soql.no-such-column","errorCode":"query.soql.no-such-column","data":{"data":{"column":"$start_day$","dataset":"alpha.34796","position":{"row":1,"column":38,"line":"SELECT * WHERE `applieddate` BETWEEN `$start_day$` AND `$datetoday$` ORDER BY `applieddate` DESC NULL FIRST\n

I would have expected the two tokens to be replaced with the date, since this is coming back from the endpoint into the index. Any ideas why the python wouldn't have executed?

I saved it into the tokens.py file, and then I restarted Splunk for good measure in case it needed to be loaded in an include or something.

Your ideas are appreciated.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Mon, 20 Jul 2015 15:27:39 GMT

I just noticed your comment that the tokens don't work in the URL arguments.

I was looking through the rest.py code. It seems like we could call the replaceTokens function for the url_args. As you know this "else" is near the end of the code.

I tried to hack a mod in here, but my python skills are weak. This didn't work, as a second "do_run", how would you suggest I could hack this in?

    else:
    config = get_input_config()
        original_endpoint=config.get("endpoint")
        #token replacement
        endpoint_list = replaceTokens(original_endpoint)
        #token replacement url args
        original_url_args=config.get("url_args")
        url_arg_list = replaceTokens(original_url_args)

        sequential_mode=int(config.get("sequential_mode",0))

        if bool(sequential_mode):
            do_run(config,endpoint_list)
         #token for url args
            do_run(config,url_arg_list)
        else:  #parallel mode
            for endpoint in endpoint_list:
                requester = threading.Thread(target=do_run, args=(config,[endpo$
                requester.start()

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

Damien_Dallimor — Mon, 20 Jul 2015 16:09:24 GMT

If you want to token replace url arguments , then encode them into the endpoint url parameter rather than declaring them in the url arguments parameter

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Mon, 20 Jul 2015 16:14:12 GMT

Thanks I'll give it a try

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

efullman — Mon, 20 Jul 2015 16:40:22 GMT

Thanks that worked as expected. I think adding token replacement in the other terms besides the endpoint would be a good improvement for the future.

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

upmangaurav — Wed, 09 Sep 2015 05:14:41 GMT

Hi Damien/efullman, Can you guys please drop a hint about where to declare these variable tokens specified inside Endpoint URL. I have a few tokens inserted into the endpoint URL which I want to be fetched from some python class. But I can't seem to figure out where to put/insert that python code in the /rest_ta/bin directory?

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

Damien_Dallimor — Wed, 09 Sep 2015 07:42:45 GMT

Have a look at the token substitution section in the documentation.

rest_ta/bin/tokens.py

Re: REST API Modular Input - Tokens in URL (Exit Status 1)

upmangaurav — Thu, 10 Sep 2015 11:53:57 GMT

Thanks Damien, That did it!