This is an odd acceleration behavior that has us stumped... If some of you worked with Qualys Technology Add-on before, Qualys dump their knowledge base into a CSV file which we converted to kvStore ...
See more...
This is an odd acceleration behavior that has us stumped... If some of you worked with Qualys Technology Add-on before, Qualys dump their knowledge base into a CSV file which we converted to kvStore with the following collections.conf accelerations enabled - The knowledge base has approx. 137,000 rows of about 20 columns. [qualys_kb_kvstore]
accelerated_fields.QID_accel = {"QID": 1}
replicate = true Then if you were to run the following query with lookup local= true and local=false (default). According to Job Inspector there was no real difference between lookup on search head vs. the indexers. Without the lookup command, the query takes 3 seconds to complete over 17 million events. With lookup added, it takes an extra 165 seconds for some reason with the accelerators turned on. index=<removed> (sourcetype="qualys:hostDetection" OR sourcetype="qualys_vm_detection") "HOSTVULN"
| fields _time HOST_ID QID
| stats count by HOST_ID, QID
| lookup qualys_kb_kvstore QID AS QID OUTPUTNEW PATCHABLE
| where PATCHABLE="YES"
| stats dc(HOST_ID) ```Number of patchable hosts!``` An idea I am going to try is to add PATCHABLE as another accelerated field and see if that changes. This change will require me to wait until tomorrow. accelerated_fields.QID_accel = {"QID": 1, "PATCHABLE": 1} Is there something we're missing to help avoid the lookup taking extra 2-3 minutes?