Splunk Dev

Custom command - Getinfo probe failed

drebai
Explorer

Hi!
I created a custom command for a comparison between json.
The steps to create it:

0) created fieldcompare folder in etc/apps (with bin, metadata and default subfolders)

1) fieldcompare.py (in etc/apps/fieldcompare/bin)

# custom command

import splunk.Intersplunk  
import json
from splunklib.searchcommands import \
     dispatch, StreamingCommand, Configuration, Option, validators


 # long def function (searchleveljson) that looks for things in jsons
 #etc etc

(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()

if isgetinfo:
     # streaming, generating, retevs, reqsop, preop
    splunk.Intersplunk.outputInfo(True, False, False, False, None)


(results, dummyresults, settings) = splunk.Intersplunk.getOrganizedResults()

field1 = kwargs.get("field1", "field1")
field2 = kwargs.get("field2", "field2")

try:
    for result in results:
        try:
            j1 = json.load(field1)
            j2 = json.load(field2)
        except KeyError:
             # If either field is missing, simply ignore
            continue

        resultKey = list()
        resultValues = list()
        searchleveljson(j1,j2,resultKey,resultValues)

        result["mismatched keys"]=json.dumps(resultKey)
        result["Value diff"]=json.dumps(resultValues)


splunk.Intersplunk.outputResults(results)

2) commands.conf (in etc/apps/fieldcompare/default)

[fieldcompare]
filename = fieldcompare.py
supports_getinfo = true

3) default.meta (in etc/apps/fieldcompare/metadata)

 [commands/fieldcompare]
 access = read : [ * ], write : [ admin ]
 export = system

 [scripts/fieldcompare.py]
 access = read : [ * ], write : [ admin ]
 export = system

4) copy splunklib (with modularinput and searchcommands) in etc/apps/fieldcompare/bin from etc\apps\framework\contrib\splunk-sdk-python

5) Restart splunk

6) index=* etc etc | table expected actual | fieldcompare field1=expected field2=actual

but I get an error that says:

Error in 'script': Getinfo probe failed for external search command 'fieldcompare'

What am I doing wrong?

Thanks,
have a good day.

Debora

0 Karma
1 Solution

drebai
Explorer

Intersplunk solution:

apps/fieldcompare/bin/fieldcompare.py:

# custom command script

import splunk.Intersplunk  
import json
import sys

def is_json(myjson):
    return type(myjson) == type({})

#found difference between two jsons 
def searchleveljson(json1,json2,resultKey,resultValues):
    keys=set(json1.keys())
    keys2=set(json2.keys())

    keys.update(keys2)
    for json_event in keys:
        if json_event not in json1:
            resultKey.append(json_event)
            resultValues.append("actual."+json_event+"."+json2[json_event])
            continue;
        if json_event not in json2:
            resultKey.append(json_event)
            resultValues.append("expected."+json_event+"."+json1[json_event])
            continue;
        v1 = json1[json_event]
        v2 = json2[json_event]
        if v1 != v2:
            if is_json(v1) and is_json(v2):
                searchleveljson(v1,v2,resultKey,resultValues)
            else:
                resultKey.append(json_event)
                diff = "actual."+json_event+"="+json2[json_event]
                diff+= " "
                diff+= "expected."+json_event+"="+json1[json_event]
                resultValues.append(diff)

(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()

if isgetinfo:
    # streaming, generating, retevs, reqsop, preop
    splunk.Intersplunk.outputInfo(True, False, False, False, None)


results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()

field1 = kwargs.get("json1", "field1")
field2 = kwargs.get("json2", "field2")


for result in results:
    try:
        #corrected errors for converting string to json
        #json.load(fieldx) -> json.loads(result[fieldx])
        j1 = json.loads(result[field1])
        j2 = json.loads(result[field2])
    except ValueError:
        pass

    resultKey = list()
    resultValues = list()
    searchleveljson(j1,j2,resultKey,resultValues)

    result["mismatched_keys"]=json.dumps(resultKey)
    result["value_diff"]=json.dumps(resultValues)


splunk.Intersplunk.outputResults(results)

apps/fieldcompare/default/custom.conf:

[fieldcompare]
filename = fieldcompare.py
streaming=true

apps/fieldcompare/metadata/default.meta:

  [commands/fieldcompare]
  access = read : [ * ], write : [ admin ]
  export = system


 [scripts/fieldcompare.py]
 access = read : [ * ], write : [ admin ]
 export = system

I deleted all other folders (spunk and splunklib)

Restart splunk.

| table column1 column2 | fieldcompare __EXECUTE__ json1=column1 json2=column2

the result is a table with the following columns:
column1 | column2 | mismatched_keys | value_diff

mismatched_keys contains all different keys
value_diff contains all different value

View solution in original post

0 Karma

drebai
Explorer

Intersplunk solution:

apps/fieldcompare/bin/fieldcompare.py:

# custom command script

import splunk.Intersplunk  
import json
import sys

def is_json(myjson):
    return type(myjson) == type({})

#found difference between two jsons 
def searchleveljson(json1,json2,resultKey,resultValues):
    keys=set(json1.keys())
    keys2=set(json2.keys())

    keys.update(keys2)
    for json_event in keys:
        if json_event not in json1:
            resultKey.append(json_event)
            resultValues.append("actual."+json_event+"."+json2[json_event])
            continue;
        if json_event not in json2:
            resultKey.append(json_event)
            resultValues.append("expected."+json_event+"."+json1[json_event])
            continue;
        v1 = json1[json_event]
        v2 = json2[json_event]
        if v1 != v2:
            if is_json(v1) and is_json(v2):
                searchleveljson(v1,v2,resultKey,resultValues)
            else:
                resultKey.append(json_event)
                diff = "actual."+json_event+"="+json2[json_event]
                diff+= " "
                diff+= "expected."+json_event+"="+json1[json_event]
                resultValues.append(diff)

(isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv)
args, kwargs = splunk.Intersplunk.getKeywordsAndOptions()

if isgetinfo:
    # streaming, generating, retevs, reqsop, preop
    splunk.Intersplunk.outputInfo(True, False, False, False, None)


results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()

field1 = kwargs.get("json1", "field1")
field2 = kwargs.get("json2", "field2")


for result in results:
    try:
        #corrected errors for converting string to json
        #json.load(fieldx) -> json.loads(result[fieldx])
        j1 = json.loads(result[field1])
        j2 = json.loads(result[field2])
    except ValueError:
        pass

    resultKey = list()
    resultValues = list()
    searchleveljson(j1,j2,resultKey,resultValues)

    result["mismatched_keys"]=json.dumps(resultKey)
    result["value_diff"]=json.dumps(resultValues)


splunk.Intersplunk.outputResults(results)

apps/fieldcompare/default/custom.conf:

[fieldcompare]
filename = fieldcompare.py
streaming=true

apps/fieldcompare/metadata/default.meta:

  [commands/fieldcompare]
  access = read : [ * ], write : [ admin ]
  export = system


 [scripts/fieldcompare.py]
 access = read : [ * ], write : [ admin ]
 export = system

I deleted all other folders (spunk and splunklib)

Restart splunk.

| table column1 column2 | fieldcompare __EXECUTE__ json1=column1 json2=column2

the result is a table with the following columns:
column1 | column2 | mismatched_keys | value_diff

mismatched_keys contains all different keys
value_diff contains all different value

0 Karma

hardikJsheth
Motivator

Are you getting any error when you start your splunk server? Any compilation error in your customcommand will get reported only at starting of Splunk server.

Also you can check https://splunkbase.splunk.com/app/3447/. It has different way of writing streaming command.

0 Karma

drebai
Explorer

it doesn't find any errors at startup.
This is right method to use intersplunk?

0 Karma

micahkemp
Champion

I would suggest moving to the SDK (referenced in your script as splunklib but not actually used). Check out the filter.py example that comes with the SDK for guidance.

0 Karma

hardikJsheth
Motivator

Yes use of intersplunk is correct.

Few things I just noticed from your code. You are creating tupple when fetching results. You should update that query as follows:

results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()

When you return result, it should be list of directories.

0 Karma

drebai
Explorer

I tried to create the custom command in both ways.

Intersplunk way:
I added "import sys" and removed import from splunklib.
From splunk search page I read an error that asks to add GETINFO or EXECUTE
I changed the command to:
| fieldcompare EXECUTE field1=x field2=y
but I get a generic error 1.
I add getOrganizedResults before outputResults but nothing has changed.
I would just like to pass two parameters and get two new ones for each line but it seems more complicated than expected.

SDK way:
I followed the documentation.
I started from SDK templates, taking inspiration from the examples and I created a class that extends a GeneratingCommand but the error is always the same:

Error in 'script': Getinfo probe failed for external search command 'jsoncompare'

0 Karma

drebai
Explorer

Thanks 🙂
I solved the problem with Intersplunk!

0 Karma

micahkemp
Champion

Please post your solution as an answer and accept that answer so that others may benefit from your findings.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...