I'm using the below search to grab a list of tag_values from one index and use it as a subsearch on another index. I'm finding not all events are getting picked up though. The subsearch returns 140 results so it's not a limitation on that end. With the subsearch I don't pick up all the messages I'm looking for for each run (e.g. I get 7 results returned for "DVT ready" but there should be a message for each). Is the event dropping related to how many events are being searched in the pipeline_logs index? This is being run over the past week also ((index=pipeline_logs AND (geniaComplete.flag OR "DVT ready" OR "acap branch path setup" OR "oc-cal job" OR "downloading raw data" OR "oc-cal ACAP processing" OR "Multichunk processing complete" OR "annotations upload to GCS" OR "SGE driver started" OR "transfer complete for all banks")) [search index=cumulus1 source=mysql-runs sourcetype=run_analysis AND analysis_type=reanalysis NOT pct_cells_sampled=10.0 NOT run_group="*HTP*" | eval tag_value=mvindex(split(file_name,"."),1) | table tag_value ]) ... View more

I currently own a 10GB daily indexing license. A few days ago I exceeded the indexing amount, however, none of my indexes saw as big a jump as should have occurred. After checking the details I found that supposedly 15.43 GB was indexed into a single index (called spore_1), however when I go to manage indexes the index only contains 1.08GB of data. Another issue arose today where I was issued a warning even though my current license usage is only at 4.341 GB. What is causing this disparity in both cases? ... View more

Here is the query I am currently using. The idea is to find the duration of the event within the subsearch then run a join in order to retain the _time aspect of the BEGIN messages for use as the starting point in the timeline visualization. The query is set to produce a timeline visualization for whatever "abcd" is. It works fine for general use but runs into a problem in a specific situation. index=default source=mysql-pipe sourcetype=pipeline_logs AND run_id="abcd" AND msg_type!="COMPLETE" | join type=left message [search index=geniachip source=mysql-pipe sourcetype=pipeline_logs AND run-id="abcd" | eval time=strptime(time_at,"%Y-%m-%d %H:%M:%S") | chart values(time) by message, msg_type | eval TimeDiff = (COMPLETE-BEGIN) * 1000] | fillnull value=0 TimeDiff | table _time message TimeDiff The table output looks like this (when adding COMPLETE and BEGIN to the table part of the query) _time message COMPLETE BEGIN duration 2016-12-11 14:35:16 bank02 per-node-processing 1481537943.000000 1481510388.000000 15271000.000 In certain cases the event contained in the message occurs twice during the run "abcd" meaning that there are two timestamps as shown below. If it occurs twice it means that either both times it completed successfully or one failed to complete while the other didn't. In this situation the eval duration does not function correctly and the column doesn't contain values. _time message COMPLETE BEGIN duration 2016-12-11 14:35:16 bank02 per-node-processing 1481537943.000000 1481510388.000000 1481524516.000000 2016-12-11 10:42:49 bank03 per-node-conversion 1481524291.000000 1481537710.000000 1481510569.000000 1481524996.000000 Is there a way to break up the two timestamps and essentially create two different events? Building off the previous table it would look like this instead _time message COMPLETE BEGIN duration 2016-12-11 14:35:16 bank02 per-node-processing 1481537943.000000 1481510388.000000 15271000.000 2016-12-11 10:39:48 bank02 per-node-processing 1481510447.000000 2016-12-11 10:42:49 bank03 per-node-conversion 1481524291.000000 1481537710.000000 34442000.000 2016-12-11 10:43:52 bank03 per-node-conversion 1481510569.000000 1481524996.000000 34694000.000 ... View more