Splunk Search

Outputlookup to a kvstore does not write results

alexc
New Member

Hello all! This will be a doozy, so get ready. We are running a search with tstats generated results,  from various troubleshooting we simplified it to the following

 

 

| tstats count  by host
| rename host as hostname
| outputlookup some_kvstore

 

 

The config of the kvstore is as follows:

 

 

# collections.conf
[some_kvstore]
field.hostname = string

 

 

 

 

# transforms.conf
[some_kvstore]
collection = some_kvstore
external_type = kvstore
fields_list = hostname

 

 

When you run the first 2 lines of the SPL, you will get quite a few results, as it queries the internal db for hosts and retrieves a count of their logs. After you add the outputlookup command, it removes all your results and will not add them to the kvstore. 

As my coworker found, there is a way to write the results to the kvstore after all, however the SPL for that is quite cursed, as it involves joining the original search back in, but the new results will be written to the kvstore.

 

 

| tstats count by host 
| rename host as hostname
| table hostname
| join hostname [ tstats count by host | rename host as hostname] 
| outputlookup some_kvstore 

 

 

 

As far as I aware, 9.1.2, 9.0.6, and latest verisions of cloud have this issue even as fresh installs of Splunk, however it does work on an 8.2.1 and 7.3.3 systems (dont ask). The Splunk user owns everything in the Splunkdir so there is no problem with writing to any files, the kvstore permissions are global, and any user can read or write to it.

So after several hours of troubleshooting, we are stumped here and not sure where we should look next. Changing to a csv is unfortunately not an option.

 

Things we have tried so far, that i can remember:

  • Completely fresh installs of Splunk
  • Cleaning the kvstore via `splunk clean kvstore -local`
  • Outputting to a csv (works)
  • Using makeresults to create the fields manually and add to the kvstore (works)
  • Using the noop command to disable all search optimization 
  • Writing to the kvstore via API (works)
  • Reading data from the kvstore via inputlookup (works)
  • Modifying an entry in the kvstore via the lookup editor app (works)
  • Testing with all search modes (fast, smart, verbose)
Labels (1)
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...