Splunk Search

What is the procedure to build your own Splunk (search related) function?

Builder

I have heard that this is possible - please correct me if I am wrong.

Firstly, the reason I want to do this. We index a large volume of financial logs, which are in the Financial Information eXchange (FIX) format. These are not really in an easily human readable format as they contain a bunch of numeric codes for fields and values, so I am trying to get Splunk to translate these logs so when my users search for them, they can understand them without having to reference their FIX documentation.

As I imagine this is quite a common problem, I'm going to ask another question about whether anyone has solved this problem already. In the meantime, knowing how to create my own search functions would be useful for me and others anyway, and I couldn't find instructions in the documentation.

Search/replace can easily be done by piping the search to "rex" sed mode. My FIX guys have selected a "top 100" translations they want, which means that the rex command (while it does actually work) is quite an inelegant way to do it, since it is about 40 lines long.

I'd like to create my own custom function "fixtranslate" using python, so I could encapsulate this search/replace inside the function. I would use Splunk web to run the search, pipe it to fixtranslate (passing raw search results to the script) which would do the search replace, and pass the modified results back to Splunk web to display.

How do I do this?

0 Karma
1 Solution

Splunk Employee
Splunk Employee

Please see my answer to your other question, as it might not be necessary to write a script as this could possibly be handled by existing Splunk functions.

But this doc: http://docs.splunk.com/Documentation/Splunk/5.0/Search/Aboutcustomsearchcommands describes how to write custom external search commands. There are also several out-of-the-box and examples scripts included with Splunk.

You should also look into how to write custom lookup scripts, which are similar but potentially more efficient and integrate into your search in a slightly different way.

View solution in original post

Splunk Employee
Splunk Employee

Please see my answer to your other question, as it might not be necessary to write a script as this could possibly be handled by existing Splunk functions.

But this doc: http://docs.splunk.com/Documentation/Splunk/5.0/Search/Aboutcustomsearchcommands describes how to write custom external search commands. There are also several out-of-the-box and examples scripts included with Splunk.

You should also look into how to write custom lookup scripts, which are similar but potentially more efficient and integrate into your search in a slightly different way.

View solution in original post

Builder

Solution to original problem with FIX logs, using a custom search command, can be found here: http://answers.splunk.com/questions/887/has-anyone-got-a-method-for-decoding-fix-financial-format-lo...

0 Karma

Builder

Looks like I would still need a separate custom lookup for each field, and since there are so many potential fields, this will be complex to configure and difficult to maintain. Custom search command still looks the most promising.

0 Karma

Builder

Dang, thats it. I didn't see the documentation as was looking in the wrong place (Developer manual rather than Search manual). Thanks I'll take a look. I will look into a custom lookup script as well.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!