Custom Search command works in search but not in dashboard

[adamhoskins]

I have created a custom search command to decode a hexadecimal field in IoT messages. It works fine when used from a simple search, but if I try to then use the search in a dashboard panel no data is returned. I can't think why this would be the case. The search command is in its own app which has been distributed to the search head and indexer, and has global accessibility. The command is a streaming command which takes a hexadecimal field as input and outputs fields and values based on the hexadecimal value

Thanks

Adam

[adamhoskins]

Here's an example message from a device. The search command decodes phyPayload:

{

    "id": "5f44141360bac5000165f58e",

    "devEui": "redacted",

    "date": 1598297107124,

    "direction": "Uplink",

    "type": "UnconfirmedDataUp",

    "phyPayload": "40d267000280d5131147cdc99ff4beaeeda9e9",

    "radioParameters": [

        {

            "freq": 867.9,

            "modulation": "LORA",

            "dataRate": "SF7BW125",

            "codingRate": "4/5"

        }

    ],

    "ack": false,

    "fCnt": 5077,

    "macCommand": null,

    "fPort": 17,

    "gwInfo": [

        {

            "gwEui": "redacted",

            "rfRegion": null,

            "rssi": -91,

            "snr": 10.5,

            "latitude": null,

            "longitude": null,

            "altitude": null,

            "channel": null,

            "radioId": null,

            "rssis": null,

            "rssisd": null,

            "fineTimestamp": null,

            "antenna": null,

            "frequencyOffset": null

        },

        {

            "gwEui": "redacted",

            "rfRegion": null,

            "rssi": -46,

            "snr": 8.5,

            "latitude": null,

            "longitude": null,

            "altitude": null,

            "channel": null,

            "radioId": null,

            "rssis": null,

            "rssisd": null,

            "fineTimestamp": null,

            "antenna": null,

            "frequencyOffset": null

        }

    ],

    "delayed": false,

    "classB": false,

    "encodingType": "HEXA"

}

The search I am testing just specifies the index, and pipes to the custom search command. It returns events when run as a search 

[ITWhisperer]

Still not clear what your search is. However, can you use spath to extract the phyPayload

...
| spath output=phypayload path="phyPayload"
...

[adamhoskins]

I've ingested the JSON data using indexed extractions, so phyPayload appears as an event field. The custom search command is a python script that I developed using the python SDK/splunklib . The custom search command is called abeeway_decoder, so my search is just

index=myindex | abeeway_decoder

I don't think it's possible to extract the payload using "standard" search commands, but I may be wrong. The Python script converts the hex string into a byte array and then does does some binary operations on the byte values. Is this possible?

[ITWhisperer]

Splitting into bytes is easy enough to do

...
| rex field=phyPayload "(?<byte1>\S{2})(?<byte2>\S{2})...

Converting to numbers easy enough to do

...
| eval b1=tonumber(byte1,16)
| eval b2=tonumber(byte2,16)
...

 Bitwise operations  apart from shift (multiply or divide by powers of 2 and modulus) not so easy, so it depends what operations you need and whether they can be turned into maths equations.

[adamhoskins]

Ok thanks, that's interesting. It might be an option, but I was wondering if there's a way to get the python script working (I inherited the majority of the code needed to do the decoding so would rather not recreate the logic if I can avoid it)

I've had a dig around and if put localop in between specifying the index and the custom search command then it does work in dashboards. So presumably the custom search is being executed on the search head. Any idea how to find out why it won't work on our indexer?

Thanks

Adam

[adamhoskins]

I seem to have solved the issue by piping the data to a fields command before piping to the custom search command. Not sure why that has made a difference but it is working now

Thanks for your help

Adam

[ITWhisperer]

Please can you provide some sample data and the search(es) you have tried