Hey folks, I suspect this is more of a Jenkins problem than a Splunk problem, but I figured I'd ask here anyway. Our Jenkins instances have the 'Splunk App for Jenkins' installed and set to send all console output from all jobs to Splunk on the "text::jenkins" sourcetype. Recently though I've noticed that there seem to be more events in Splunk than there should be. For example if I do a search to look for a specific job's console output , like index=jenkins_console sourcetype="text:jenkins" earliest=-5d@d latest=now source=source_of_job_I_Want | stats values(_raw) by _time I see:  From Splunk query   But if I go directly to Jenkins and look at the console output for this job I see:  From Jenkins Console This is an issue because we have an alert for the "Cannot contact X: java.lang.InterruptedException" String to help detect agent failures, but it's confusing when that alert goes off but then we look at the console and don't see that message.  Anyone come across anything similar?  ... View more

I have a search that gets events related to procedures from the past week and organizes them into days. I also have a lookup table which holds the values of the procedure names I want to look for. This lookup table data has no date values, unlike my splunk data,  since I want to compare it against every day represented.  Search:  (index=app host=host1sourcetype=st1) OR (index=chub source=s2) earliest=-1w@w latest=now [|inputlookup chubDashboardProcedures.csv | fields 1.0_Procedures | rename 1.0_Procedures as search|format "" "(" "" ")" "OR" ""] | fields Procedure_Name,Process_Name,Activity_Code, UpdatedDate | eval Procedure_Name=coalesce(Process_Name, Procedure_Name) | eval update = strptime( UpdatedDate, "%Y-%m-%d %H:%M:%S") | eval Day = relative_time(update,"@d") - if((tonumber(strftime(update, "%H%M")) < 1400), (24*60*60), 0) | dedup Procedure_Name Day | append [|inputlookup chubDashboardProcedures.csv|fields 1.0_Procedures, UpdatedDate| rename 1.0_Procedures as Procedure_Name_New| eval from="lookup"] | stats values(Procedure_Name_New) as lookup_procedures values(from) as from count(Procedure_Name) as fromcount values(Procedure_Name) as Procedure_Name_List values(eval(if(Activity_Code !="2000", Procedure_Name,null()))) as Failures by Day | mvexpand lookup_procedures | eval missing= if(match(Procedure_Name_List,lookup_procedures), "SAME", lookup_procedures) | sort - Day | eval Success_Percent = round(((Success_Count/Procedure_Count)*100),2) | eval Day = strftime(Day, "%F") | table Procedure_Name_List, lookup_procedures Failures missing  The lookup table values are there and I can compare them to a list of procedures that haven't been grouped into dates, but as soon as I group into dates I can't compare them anymore. I want to display events that are found in the lookup table but not the events for each day.  ... View more