This probably is partially covered by
https://docs.splunk.com/Documentation/Splunk/6.5.2/ReleaseNotes/Workaroundforsearchoptimizationissues
and
https://docs.splunk.com/Documentation/Splunk/6.5.2/ReleaseNotes/6.5.2
but in my particular case (after upgrading to splunk 6.5.2), searches are fundamentally broken. E.g. given sample events like
t=1487169791, rn=315827, part=LArBarrelPS, uname=nikiforo, msgID=rc::ApplicationSignaled, host=pc-tdq-onl-05.cern.ch, app=RootController, sev=WARNING, text="Application "CHIP" on host "pc-tdq-onl-06.cern.ch" died while exiting on signal 9. Logs are "/logs/tdaq-06-01-01/LArBarrelPS/CHIP_pc-tdq-onl-06.cern.ch_1487169011.out/err".", context="PACKAGE_NAME: RunControl. FILE_NAME: ../src/Controller/ApplicationController.cxx. FUNCTION_NAME: daq::rc::ApplicationController::pmgCallback(const std::weak_ptr<daq::rc::ApplicationController>&, daq::pmg::Process*, void*)::<lambda()>. LINE_NUMBER: 182. DATE_TIME: 1487169791.", params="application: CHIP. errStream: /logs/tdaq-06-01-01/LArBarrelPS/CHIP_pc-tdq-onl-06.cern.ch_1487169011.err. hostName: pc-tdq-onl-06.cern.ch. onExit: 1. outStream: /logs/tdaq-06-01-01/LArBarrelPS/CHIP_pc-tdq-onl-06.cern.ch_1487169011.out. signal: 9. ", quals="RunControl ", chained=0, gh=1414542713
simple search
index=ers | search sev=WARNING
returns no results, whereas the suggested workaround
index=ers | search sev=WARNING | noop search_optimization=false
works as before with splunk-6.4. Basically search by any non-indexed field is broken. I have some fields indexed at insertion and also field extraction configured for my events, may be there is some correlation.
sample data
configuration files (props,transforms,fields).conf
... View more