topic Re: Custom Search command works in search but not in dashboard in Splunk Dev

Custom Search command works in search but not in dashboard

adamhoskins — Mon, 07 Sep 2020 14:24:42 GMT

I have created a custom search command to decode a hexadecimal field in IoT messages. It works fine when used from a simple search, but if I try to then use the search in a dashboard panel no data is returned. I can't think why this would be the case. The search command is in its own app which has been distributed to the search head and indexer, and has global accessibility. The command is a streaming command which takes a hexadecimal field as input and outputs fields and values based on the hexadecimal value

Thanks

Adam

Re: Custom Search command works in search but not in dashboard

ITWhisperer — Mon, 07 Sep 2020 14:27:04 GMT

Please can you provide some sample data and the search(es) you have tried

Re: Custom Search command works in search but not in dashboard

adamhoskins — Mon, 07 Sep 2020 14:37:55 GMT

Here's an example message from a device. The search command decodes phyPayload:

{

"id": "5f44141360bac5000165f58e",

"devEui": "redacted",

"date": 1598297107124,

"direction": "Uplink",

"type": "UnconfirmedDataUp",

"phyPayload": "40d267000280d5131147cdc99ff4beaeeda9e9",

"radioParameters": [

{

"freq": 867.9,

"modulation": "LORA",

"dataRate": "SF7BW125",

"codingRate": "4/5"

}

"ack": false,

"fCnt": 5077,

"macCommand": null,

"fPort": 17,

"gwInfo": [

{

"gwEui": "redacted",

"rfRegion": null,

"rssi": -91,

"snr": 10.5,

"latitude": null,

"longitude": null,

"altitude": null,

"channel": null,

"radioId": null,

"rssis": null,

"rssisd": null,

"fineTimestamp": null,

"antenna": null,

"frequencyOffset": null

{

"gwEui": "redacted",

"rfRegion": null,

"rssi": -46,

"snr": 8.5,

"latitude": null,

"longitude": null,

"altitude": null,

"channel": null,

"radioId": null,

"rssis": null,

"rssisd": null,

"fineTimestamp": null,

"antenna": null,

"frequencyOffset": null

}

"delayed": false,

"classB": false,

"encodingType": "HEXA"

}

The search I am testing just specifies the index, and pipes to the custom search command. It returns events when run as a search

Re: Custom Search command works in search but not in dashboard

ITWhisperer — Mon, 07 Sep 2020 14:46:06 GMT

Still not clear what your search is. However, can you use spath to extract the phyPayload

... | spath output=phypayload path="phyPayload" ...

Re: Custom Search command works in search but not in dashboard

adamhoskins — Mon, 07 Sep 2020 14:52:44 GMT

I've ingested the JSON data using indexed extractions, so phyPayload appears as an event field. The custom search command is a python script that I developed using the python SDK/splunklib . The custom search command is called abeeway_decoder, so my search is just

index=myindex | abeeway_decoder

I don't think it's possible to extract the payload using "standard" search commands, but I may be wrong. The Python script converts the hex string into a byte array and then does does some binary operations on the byte values. Is this possible?

Re: Custom Search command works in search but not in dashboard

ITWhisperer — Mon, 07 Sep 2020 15:21:09 GMT

Splitting into bytes is easy enough to do

... | rex field=phyPayload "(?<byte1>\S{2})(?<byte2>\S{2})...

Converting to numbers easy enough to do

... | eval b1=tonumber(byte1,16) | eval b2=tonumber(byte2,16) ...

Bitwise operations apart from shift (multiply or divide by powers of 2 and modulus) not so easy, so it depends what operations you need and whether they can be turned into maths equations.

Re: Custom Search command works in search but not in dashboard

adamhoskins — Mon, 07 Sep 2020 16:24:05 GMT

Ok thanks, that's interesting. It might be an option, but I was wondering if there's a way to get the python script working (I inherited the majority of the code needed to do the decoding so would rather not recreate the logic if I can avoid it)

I've had a dig around and if put localop in between specifying the index and the custom search command then it does work in dashboards. So presumably the custom search is being executed on the search head. Any idea how to find out why it won't work on our indexer?

Thanks

Adam

Re: Custom Search command works in search but not in dashboard

adamhoskins — Mon, 07 Sep 2020 18:55:17 GMT

I seem to have solved the issue by piping the data to a fields command before piping to the custom search command. Not sure why that has made a difference but it is working now

Thanks for your help

Adam