topic Re: How to call a python script from an html view? in Splunk Dev

How to call a python script from an html view?

dsollen — Mon, 01 Feb 2016 22:31:06 GMT

We have a Splunk html view. We are trying to call a python script. Currently the code looks like this (retyped since our Splunk server doesn't have web access, I tried to avoid typos):

function callScript(){

   $.ajax({
                 type: "POST",
                 url: "bin/my_script.py",
                 data: {'flag': defaultTokenModel.get("form.flag"))
                 },
                 success: function() {
                     alert("success returned");
                 }
           });
    }

This doesn't work. The success message is returned, but the actual python script isn't called.

From what I can tell, it looks like there may be a method called ajax in simplifyxml/mvc.js which is conflicting with the JQuery method, though I'm not certain that is the case.

What is the proper way to make a script call like this?

Re: How to call a python script from an html view?

alacercogitatus — Tue, 02 Feb 2016 15:22:08 GMT

This won't work as you think. Access to local folders within an app is not allowed for security and technical reasons.

When you make that POST call, you are actually trying to get a Web UI relative script! https:///en-US/app/myapp/bin/my_script.py for example. This just doesn't work.

So, you cannot directly call a python script from the web ui, it is not accessible. There is a way to do this. It is a method not fully documented, but involves creating a custom endpoint for Splunk, and then calling that endpoint.

Let me start with a quick tutorial. I'll gloss over a few things, make assumptions, drink some beer. Comment with questions. 😄

web.conf

[endpoint:my_script]

So, that is the only line you need in web.conf. This tells Splunk that you are creating a new web endpoint. The my_script definition needs to have the same name as the python file located in $APP_HOME/appserver/controllers. (Where $APP_HOME is the folder of the app you are creating this in - I'd suggest a new app, not search).

$APP_HOME/appserver/controllers/my_script.py

import logging
import os
import sys

# from splunk import AuthorizationFailed as AuthorizationFailed
import splunk.appserver.mrsparkle.controllers as controllers
import splunk.appserver.mrsparkle.lib.util as util
from splunk.appserver.mrsparkle.lib.util import make_splunkhome_path
from splunk.appserver.mrsparkle.lib.decorators import expose_page

_APPNAME = 'MyAppName'

def setup_logger(level):
    """
    Setup a logger for the REST handler.
    """

    logger = logging.getLogger('splunk.appserver.%s.controllers.my_script' % _APPNAME)
    logger.propagate = False  # Prevent the log messages from being duplicated in the python.log file
    logger.setLevel(level)
    file_handler = logging.handlers.RotatingFileHandler(
        make_splunkhome_path(['var', 'log', 'splunk', 'my_script_controller.log']), maxBytes=25000000, backupCount=5)
    formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    return logger


logger = setup_logger(logging.DEBUG)


class my_script(controllers.BaseController):
    # /custom/MyAppName/my_script/my_endpoint
    @expose_page(must_login=True, methods=['GET'])
    def my_endpoint(self, **kwargs):
        # DO YOUR THINGS WITH THE KWARGS PASSED

Whoa. WTF is that!? Python! YAY! So, now you have an endpoint enabled. Now you can call it from your HTML view.

 function callScript(){
    $.get("/custom/MyAppName/my_script/my_endpoint", 
               {'flag': defaultTokenModel.get("form.flag")) } )
                .done(function(data){
                  });
     }

Pay attention to these strings throughout this tutorial: my_script, my_endpoint, MyAppName. This all need to be changed to your specification, they are merely examples. Again, this should at least point you in the right direction on doing this call. If you have questions, please follow up here, or find me on IRC (#splunk on efnet - I'm alacer) or via the Splunk Slack channel for usergroups, just ping me @alacercogitatus.

Re: How to call a python script from an html view?

dsollen — Tue, 02 Feb 2016 17:49:34 GMT

I do have a few questions. since I already am using an app I assume I want to use $app_home/local/web.conf to make the change?

More importantly, I'm not certain I understand the format of the my_script.py example you gave; what is or isn't required to make my python script function as an endpoint. the web.conf definition says that endpoints registers a default cherrypy script, but your example doesn't import cherrypy. Is there a link that defines the format required and what the imported controller and other mrssparkle libs you used do? (I couldn't find an API for them either, my google-fu is really failing me today)

One last question, what do I need to do to ensure splunk sees my changes to the app after I do this? would 'splunk install' with an -update work?

Re: How to call a python script from an html view?

arkadyz1 — Tue, 02 Feb 2016 18:17:35 GMT

I'm pretty sure it's etc/system/local/web.conf

Re: How to call a python script from an html view?

arkadyz1 — Tue, 02 Feb 2016 18:43:24 GMT

This is an interesting example, doubly so for me since I've done something very similar but in a different way. Please take a look at this post:

Return binary data (such as images) via REST API

There, I used [expose:...] stanzas in web.conf and my additions to restmap.conf. This made my endpoint visible under /services via admin port, which is something I feel is not quite appropriate and very cumbersome. Where is your approach documented?

Re: How to call a python script from an html view?

alacercogitatus — Tue, 02 Feb 2016 18:50:42 GMT

Yes, use the $APP_HOME/local/web.conf if you didn't make the app, but use $APP_HOME/default/web.conf if this is an app you created. DO NOT USE etc/system/local!

In the example script, just copy and paste that entire thing into the controllers folder. Your bin/my_script.py steps go into the section under the def at # DO YOUR THINGS WITH THE KWARGS PASSED.

Don't need to import cherrypy for this example, because I'm not using anything from cherrypy. The cherrypy in the documentation is simply stating that this endpoint is exposed via the Splunk Web Server.

You will need to do a full restart (on the search head) for these changes to be picked up. A simple update won't work.

Here is about as close to documentation as you will get: http://docs.splunk.com/Documentation/Splunk/6.3.2/ModuleRef/Extensiondictionary

Re: How to call a python script from an html view?

alacercogitatus — Tue, 02 Feb 2016 18:50:59 GMT

Yeah, don't do that.

Re: How to call a python script from an html view?

alacercogitatus — Tue, 02 Feb 2016 20:55:57 GMT

It's not really, tbh. It's from years of Splunk App Dev and experimentation with other Apps and finding what works.

Re: How to call a python script from an html view?

dsollen — Wed, 03 Feb 2016 15:29:10 GMT

just so you know the post you mentioned isn't linked, and a quick search doesn't reveal it; making it hard to check it out 🙂 My good friend google says the link is https://answers.splunk.com/answers/306952/return-binary-data-such-as-images-via-rest-api-or.html

and yes I wish there was better documentation. I feel blind following suggestions here without full understanding of them. but I can't find any documentation of these things even after they are explained to me, so maybe the documentation doesn't entirely exist?

Re: How to call a python script from an html view?

arkadyz1 — Wed, 03 Feb 2016 15:51:06 GMT

Your friend google was correct, as usual :). From alacer's reply to my comment I can see that the documentation is indeed scarce. As I got used to saying in my company (not only about Splunk), "underdocumented" is a huge understatement :). Gone are the companies like DEC which had really exhaustive User Guides and Reference Manuals...

Re: How to call a python script from an html view?

dsollen — Wed, 03 Feb 2016 19:02:36 GMT

Just getting back to this. I did this but it didn't quite work for me, I'm getting a 500 error when I try to post (my original example was a post not a get, I went ahead and coverted over to POSTS). I think the problem is that I have the wrong URL to post to...

Currently my URL to the HTML page is something like

hostname/splunk/en-us/app/MyAppName/html_file

your example says that the url starts at custom, but I wasn't expecting custom, and I don't know what that URL is relative to. If I want to connect directly to my rest url to verify something is happening what URL should I be using?

Sorry to ask such a basic question, this is the sort of thing I shouldn't need help with. but I feel blind here, not sure where any documentation is to check on my own; I've been looking lol.

Re: How to call a python script from an html view?

alacercogitatus — Tue, 16 Feb 2016 18:44:45 GMT

So to construct the URL, you need 3 things. 1) Name of your App 2) Name of the Endpoint 3) name of the function.

Then it goes such like: /custom/APPNAME/ENDPOINT/FUNCTION The /custom denotes that this is a custom endpoint and is required for this to work. So if your "script" (located in $APP_HOME/appserver/controllers/) is called html_file.py, and the function in that file is called post_this , then your URL in the HTML dashboard will be /custom/MyAppName/html_file/post_this. Done.

Re: How to call a python script from an html view?

earakam — Wed, 23 Mar 2016 06:15:52 GMT

Hi,
I tried the above but in my python, i have the error below:

ImportError: No module named cherrypy

isn't this cherrypy packaged with splunk as default? or do i need to install it?

Re: How to call a python script from an html view?

alacercogitatus — Wed, 23 Mar 2016 11:22:47 GMT

System installed python will not have that module, and I think that is what you tried to do. Use Splunk's python instead: $SPLUNK_HOME/bin/splunk cmd python myscript.py

Re: How to call a python script from an html view?

earakam — Thu, 24 Mar 2016 00:00:43 GMT

Hi alacercogitatus!

I really appreciate your quick response. Thank you.
I will try that! but which folder should myscript.py be in to do that?
I'm wondering cause i am trying ajax, aswell, to call python script but I don't know what kind of 'URL' i should write for that.

Re: How to call a python script from an html view?

kasu_praveen — Thu, 05 May 2016 14:40:32 GMT

@alacercogitatus: Thanks for posting this. I was actually looking for this.

I tried the way you mentioned above, but seems to be something went wrong, I am seeing 404 error in my browser console and my HTML view shows nothing.

http://localhost:8000/en-US/custom/MyAppName/my_script/my_endpoint?_=1462458361563 Failed to load resource: the server responded with a status of 404 (Not Found).

Can you help me to resolve this?

Re: How to call a python script from an html view?

alacercogitatus — Thu, 05 May 2016 14:52:17 GMT

Did you restart ? Once you make the changes to create the endpoint, you have to restart.

Re: How to call a python script from an html view?

kasu_praveen — Thu, 05 May 2016 14:53:54 GMT

I did restarted, but no luck.

Re: How to call a python script from an html view?

alacercogitatus — Thu, 05 May 2016 14:55:39 GMT

Check that the Python is compiling, ($SPLUNK_HOME/bin/splunk cmd python ./my_script.py) and then check the internal splunk logs for errors and warnings.

Re: How to call a python script from an html view?

kasu_praveen — Thu, 05 May 2016 15:11:16 GMT

Seems to be python compiling is issue, I am very new to python, have to see how this compiling stuff works.
Also, do I need to compile each time when I modified python code? will that won't work, If I hit HTML view directly in browser?