I'm using the below search to grab a list of tag_values from one index and use it as a subsearch on another index. I'm finding not all events are getting picked up though. The subsearch returns 140 results so it's not a limitation on that end. With the subsearch I don't pick up all the messages I'm looking for for each run (e.g. I get 7 results returned for "DVT ready" but there should be a message for each). Is the event dropping related to how many events are being searched in the pipeline_logs index? This is being run over the past week also
((index=pipeline_logs AND (geniaComplete.flag OR "DVT ready" OR "acap branch path setup" OR "oc-cal job" OR "downloading raw data" OR "oc-cal ACAP processing" OR "Multichunk processing complete" OR "annotations upload to GCS" OR "SGE driver started" OR "transfer complete for all banks")) [search index=cumulus1 source=mysql-runs sourcetype=run_analysis AND analysis_type=reanalysis NOT pct_cells_sampled=10.0 NOT run_group="*HTP*" | eval tag_value=mvindex(split(file_name,"."),1) | table tag_value ])
The subsearch have limitation on the execution time as well, apart from number of rows returned. (see link below). It could be possible that the subsearch is auto finalized due to longer processing time. Do you see any message in the job dropdown (below search bar) regarding your subsearch?
It doesn't seem to be an execution time limit. Running the entire search only takes 10 seconds.
My title may have been off. Events may not be being dropped during the subsearch but on the entire search. For some tag_values I get 2/10 messages even though all messages exist