We have a daily run which creates logs which are ingested into Splunk.
I've created a dashboard in Splunk which runs fine if I run it later in the day near the end of the run or after it is finished. All fields populate correctly.
However if I run the reports in the dashboard earlier in the day some of the fields in the tables are blank.
These blank fields show up even if I run the reports for a date range in the past using a dashboard time picker or even if I copy the SPL query out and run it as a standalone report.
I get the feeling it's something to do with the joins I've used in the queries but I can't see why as they are left outer joins and as I mentioned even if I run the reports for a date range in the past I get the blanks. It seems to be related to when I run the report itself.
For example I've just run the query below and the "LPE_Total" field is blank for 5 of the last 7 days. Also the "LPE_F_Start" and "LPE_F_End" fields are blank for one of the days. If I run this query later on today these fields will be populated correctly.
I assure you all the data is actually there right now!
Any help would be appreciated.
index=lpe | bucket _time span=1d as Day | stats earliest(_time) as LPE_Start latest(_time) as LPE_End by Day
| eval LPE_Start=strftime(LPE_Start,"%H:%M:%S")
| eval LPE_End=strftime(LPE_End,"%H:%M:%S")
| eval Day=strftime(Day,"%d/%m/%Y")
| join type=left Day [search index=mule-new State=START Service=LPE | bucket _time span=1d as Day | stats values(FileTime) as LPE_F_Start by Day | eval Day=strftime(Day,"%d/%m/%Y")]
| join type=left Day [search index=mule-new State=END Service=LPE | bucket _time span=1d as Day | stats values(FileTime) as LPE_F_End by Day | eval Day=strftime(Day,"%d/%m/%Y")]
| eval st = strptime(LPE_Start,"%H:%M:%S") | eval et = strptime(LPE_F_End,"%H:%M:%S") | eval diff = et - st | eval LPE_Total = tostring(diff, "duration")
| eval LPE_Total=replace(LPE_Total,"\.\d+","")
| fields Day LPE_Start LPE_End LPE_F_Start LPE_F_End LPE_Total