Splunk Search

Getting an intermittent error searching against a kvstore with json using inputlookup and lookup- How to fix?

donelliot
Path Finder

on splunk cloud 8.2.2202.2

issuing the command as follows I get an error one times out of four - 

 


| inputlookup append=t ethos_vulnaction_generic

Last 30 minutes
 
Error in 'inputlookup' command: External lookup table 'inputlookup' returned error code 0. Results might be incorrect.
The search job has failed due to an error. You may be able view the job in the Job Inspector.


| inputlookup append=t ethos_vulnaction_generic

restarted splunk - no luck

Not sure how to decipher job inspector - but this inconsistency - sometimes it work sometimes it doesn't is strange.

kvstore was populated with json, and lookup; does have a filter in it - NOT asset_specific = "true"

I tried removing the filter seeing if this impacted the results but I still get an error about one time in four.. 

if i do a rest query of the kvstore in json it looks  healthy to me... besides if I take this filter out I still get stability issues
        "asset_specific": true,

A cut down example of the json used to populate the record. I do refer explicitly to the field in the lookup as details.plugin_id which the lookup command seems to like... a snippet of json

{
"action_description": "zulu specific",
"asset_specific": true,
"details": {
"plugin_id": [
"153989"
]
}
}

Labels (1)
Tags (2)
0 Karma

donelliot
Path Finder

Ok found a bit more...

56 _userContext= nobody
08-19-2022 03:15:09.120 ERROR HttpClientRequest [13376 phase_1] - Caught exception while parsing HTTP reply: Unparsable gzip header in HTTP response
08-19-2022 03:15:09.120 ERROR KVServiceClient [13376 phase_1] - KVServiceClient transaction failed after 0 retries. uri = <blah>
08-19-2022 03:15:09.120 ERROR SSCInputLookup [13376 phase_1] - Failed to call KVServiceClient for Input Lookup:
08-19-2022 03:15:09.120 ERROR SearchOperator:inputcsv [13376 phase_1] - Error in 'inputlookup' command: External lookup table 'inputlookup' returned error code 0. Results might be incorrect.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...