How do you create custom modules

[hoffmandirt]

What is involved in creating custom modules? I'm looking at the existing modules and I'm not sure how all of the files work together. Take the SingleValue module for example. It has a js, py, pyo, html, css, and conf file. How are these files put together at runtime?

I am trying to modify the SingleValue module for my application. I want to remove the DIVs out of SingleValue.html, but I don't want to change the template because it will affect the other applications. So I started going down the path of creating my own SingleValue custom module that renders different HTML. I searched through the developers guide and I didn't see anything about creating custom modules.

[dbrezinski]

I recently went through the process of creating a custom module to integrate a javascript graphing library that takes a relatively complex JSON data-structure as input. Along the way there were many gotchas and digging I had to do to get things to work well. Here is a list of the most important things I found:

• create a directory for your module in $SPLUNK_HOME/etc/apps//appserver/modules

• create a configuration file for your module in the directory you created for your module. Name it .conf. It needs to have a stanza header "[module]". In the stanza, it needs to have className and superClass attributes, so the file will look something like:

[module]
className = Splunk.Module.<your module name>
superClass = Splunk.Module.<the module you inherit from>
description = This module does super cool stuff

The referenced class and super class names relate to your module's javascript class. Your module's superclass will probably be one of the classes from modules in $SPLUNK_HOME/share/splunk/search_mrsparkle/modules. For instance, a results module might inherit from Splunk.Module.DispatchingModule, which is defined in DispatchingModule.js.

• create your javascript class in a file called <your module name>.js in your module directory (same as where your conf file is located). This is one of the tricky parts. Your javascript class implements a controller pattern that ultimately inherits from Splunk.Module.AbstractModule (in $SPLUNK_HOME/share/splunk/search_mrsparkle/modules/AbstractModule.js). Read that module and any other in the inheritance chain you choose (DispatchingModule for example).

Minimally your class will need to implement the renderResults method, but you will probably want to implement getResultParams, onContextChange and possibly onJobStatusChange and onJobDone. Your js file might end up having the following structure and method signatures (but with your code in the methods and inheriting from the appropriate module):

Splunk.namespace("Module");

Splunk.Module.<your module class name> = $.klass(Splunk.Module.DispatchingModule, {
    initialize: function($super, container) {
        $super(container);
    },

    onJobDone: function(evt) {
        this.getResults();
    },

    getResultParams: function($super) {
    var params = $super();
        var context = this.getContext();
        var search = context.get("search");
        var sid = search.job.getSearchId();
        if (!sid) 
            this.logger.error(this.moduleType, "Assertion Failed. getResultParams was called, but searchId is missing from my job.")
        params.sid = sid;
        return params;
    },

    onJobStatusChange: function(event, status) {
        if (status == 'cancel') {
            this.reset();
        }
    },

    onContextChange: function() {
        if (this.haveResultParamsChanged()) {

        }
    },

    renderResults: function($super, results) {

    }

});

• If you need to do any transformation or formatting of the search results set that is difficult or impossible to do in the splunk search pipeline, you probably want to do it in python for your module. This turns out to be pretty simple once you see the template for how it is implemented. You need to create a python file in your module directory named <your module name>.py. It probably only needs to create a class that implements the generateResults method. However, the class name and parameter list need to be specific to your module and lined up with your javascript. Specifically, the python classname must be the same as your javascript classname, i.e. "class <your module class name>(module.ModuleHandler):". The parameter list for generateResults needs to include the parameters you setup on the params object in your javascript getResultParams method after the required base parameters self, host_app, client_app. In these skeleton files, the sid attribute is set on params in the javascript, so it is the fourth parameter to generateResults in this python class. These extra parameters can/need to coincide with params your module can accept from user interface xml that uses your module. A template for your py file looks like this:

  import cherrypy import controllers.module as module

  import splunk, splunk.search, splunk.util, splunk.entity import lib.util as util

  import logging logger = logging.getLogger('splunk.appserver.controllers.module.')

  class (module.ModuleHandler):

  def generateResults(self, host_app, client_app, sid):
      # assert input
      if not sid:
          raise Exception('generateResults - sid not passed!') 
      # get job
      try:
          job = splunk.search.getJob(sid, sessionKey=cherrypy.session['sessionKey'])
      except splunk.ResourceNotFound:
          return _('Invalid Job')
      if job.resultCount > 0:
          <job.results, job.resultCount contain the interesting stuff>....
          return <string>       
      return _('No results')
  

The result string returned will then be passed to your javascript renderResults method as the results parameter. JSON is a good way to move complex data between your python code and the javascript that can actually insert content into the view based on it. WARNING: the version of jQuery included in Splunk does not have jQuery.parseJSON, so you need to use the browser's native JSON.parse or your own included code.

job.results is a list of dicts, one for each result, and the keys to the dicts are the field names created in the Splunk search and the values are lists of the field values ([0].value gets you the field data in most cases).

• Optionally you can add .html and .css files that use your module name. They will be inserted and included by the xml element that calls your module (<module name="<your module name>">). If you want to include custom javascript in your module beyond the required controller methods, you can either use a script tag in your module's html file and/or add it to your module's js file. A common pattern might be to create a div in the html with an id, and use javascript included in your js and/or html file to add/manipulate content in the named div. Use the jQuery methods available in the version included in Splunk to do the required changes in the view beyond what is possible with just CSS.

You need to potentially restart splunk to get it to pickup changes to your python or go to manager then back to your app to pickup changes to your js, html or css files. Make sure you purge your browser's cache to get the changed js or css. You will bang your head on the desk if you don't 😉

Finally, the flow of all this module code looks like:

1. module gets instantiated by reference in view xml by a module element specifying its name.
2. any html and/or css files in your module will be injected into the rendered view at this point.
3. your javascript will be included in the rendered view. the controller methods in your javascript class will get called as events are pushed down from other modules, i.e. search job status changes. For example, onJobStatusChange and onJobDone are called at obvious times. onContextChange is called if parameters to the search change (the user changed the search and kicked it off), so you probably want to use this method to clear results from the view from the prior search etc.
4. once the results are ready to be generated, your python class' generateResults method is called. You need to retrieve the job, using the sid, and retrieve the results through the job. Whatever you return from this method will get passed to your javascript renderResults method. If you don't implement a python class, I am not sure what format the results object has that is passed to your javascript. You will have to figure that out.
5. your renderResults javascript method will get called and is the final stop in your module. It needs to do whatever it is with the results that you want to do. This is assuming that your module is doing a custom results renderer.

[hoffmandirt]

Thanks, the module.py file helped a lot. The path is %SPLUNK_HOME%\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib for those using 4.x on Windows.

[sideview]

Its best to package the custom module in the app if you can, rather than creating a module in $SPLUNK_HOME/share/splunk/search_mrsparkle/modules.

I think you're right - there's no documentation for this somehow. We'll fix that.

But for now, to package it within your app you put the files at

$SPLUNK_HOME/etc/app/<appname>/appserver/modules/<ModuleName>/

You need to have at least a <ModuleName>.js and a <ModuleName>.conf. (Read through some of the other conf files to get the feel for the conventions and how to set up some simple validation on your params. )

You can also have any of: <ModuleName>.css, a <ModuleName>.html file and <ModuleName>.py

(Its possible that you actually need the <ModuleName>.html and a <ModuleName>.css but I dont think so -- if you do that's a bug because the regular modules do not require them)

As far as subclassing an existing module, to subclass the behaviour of the SingleValue module without changing anything -- you'd have a JS file that looked just like this:

Splunk.Module.MyCustomSingleValue = $.klass(Splunk.Module.SingleValue, {

});

To change the .html on the other hand there's no way to just inherit the SingleValue.html template as far as I know but you can duplicate the mako template entirely.

Depending on what you change you might need to implement some of the SingleValue methods like maybe renderResults(). If you need to do that, you're js file will look more like this:

Splunk.Module.MyCustomSingleValue = $.klass(Splunk.Module.SingleValue, {
    // when $super is the first argument to an overridden function, 
    // as in other languages it is a bound method reference to the 
    // superclass implementation.
    renderResults: function($super, result) {
         // call the super if its useful / applicable.
         //$super(results)  
         ... implement your own rendering code...
    }
});

Hopefully that gives you enough info to make progress.

[hoffmandirt]

I didn't create my own python file. I created a JavaScript file that subclassed SingleValue and I created an HTML file that was a copy of the SingleValue HTML file. I'll check the logs. Thanks!

[sideview]

Interesting. Well it sounds like something in the Python is unhappy. I've used custom modules a lot recently but come to think of it I may never have used a .py file in any of them. Maybe look for anything out of the ordinary that would prevent it from importing your python, look for errors in web_service.log, or ask support for help.

[hoffmandirt]

It's not shared. I moved my module to the main modules directory and now I get this error: Splunk could not find a controller to import for the following module: "Splunk.Module.Test". Any ideas? I'm trying to extend SingleValue like you describe.

[sideview]

That's very odd. I would send in a description and a copy of the view XML and module files to support... Are you by any chance sharing that view across all apps and accessing it from a different app? That's the only thing i can think of. I've done this a couple times recently in 4.1 and it worked fine fwiw.

[hoffmandirt]

Splunk is requiring the use of an HTMLfile for the component.

[hoffmandirt]

I just checked firebug. I am getting a 404 error with the new JavaScript files. It appears that Splunk is searching the main modules directory instead of the application module directory. I'll continue to look around.

[hoffmandirt]

I followed your instructions and the module is showing up when I go to http://localhost:8000/info and select the module listing. The module is not working in the application though. I'm getting the following error: Splunk encountered the following unknown module: "Test" . The view may not load properly. Any ideas?

[Lowell]

I'm assuming that you've done some poking around in $SPLUNK_HOME/share/splunk/search_mrsparkle/modules? I think you have to do a bump to get some of those resources to reload. Try the URL: http://splunkserver:8000/en-US/_bump (not sure where I read this). Also take a look at $SPLUNK_HOME/lib/python2.6/site-packages/splunk/appserver/mrsparkle/lib/module.py. Hopefully this helps get you started; I'm also hoping for a more offical and less go-figure-it-all-out-yourself kind of answer to this. 😉