topic Re: Summary index event timestamp issue in Splunk Search

Summary index event timestamp issue

mlf — Mon, 28 Sep 2020 18:16:52 GMT

I have a search that generates 24 hours of timechart results with a 10 minute span. The search returns expected results when run manually, but when I schedule it and enable summary indexing, the timestamp Splunk assigns to the last event generated in the summary is incorrect. E.g. the following event

11/29/2014 23:50:00 -0500, search_name="XZY test", search_now=1417150020.000, info_min_time=1413517620.000, info_max_time=1417150020.000, info_search_time=1417150021.521, high="500.149698", low="416.993160", pred="458.571429"

Ends up with a timestamp of "11/29/14 11:40:00.000 PM". All the other timestamps match the timechart generated times. This effectively creates a spike followed by a gap in the summary data.

Currently running 6.1.3. Anyone seen this before?

Re: Summary index event timestamp issue

martin_mueller — Sun, 30 Nov 2014 15:15:53 GMT

What's the values of timestartpos and timeendpos for that event?

Re: Summary index event timestamp issue

mlf — Mon, 01 Dec 2014 14:58:21 GMT

timestartpos=0, timeendpos=25 for that and all other events in the series.

Re: Summary index event timestamp issue

martin_mueller — Mon, 01 Dec 2014 15:31:50 GMT

Okay, that's the entire timestamp at the beginning... weird.

Re: Summary index event timestamp issue

mlf — Mon, 28 Sep 2020 18:18:13 GMT

My though exactly. Here's some additional info:
The query the generated the even above runs every day at 23:47. I created a similar query (same structure but different data) and set it to run at 23:37. It shows a similar issue, although instead of one event with the wrong time, it generates two. All three of the following events show a timestamp of "12/1/14
11:30:00.000 PM":

12/01/2014 23:50:00 -0500, search_name="ABC volume", search_now=1417322220.000, info_min_time=1413689820.000, info_max_time=1417322220.000, info_search_time=1417322223.036, high="40.08991", low="0.42437", pred="20.257143"

12/01/2014 23:40:00 -0500, search_name="ABC volume", search_now=1417322220.000, info_min_time=1413689820.000, info_max_time=1417322220.000, info_search_time=1417322223.036, high="39.87579", low="1.72421", pred="20.800000"

12/01/2014 23:30:00 -0500, search_name="ABC volume", search_now=1417322220.000, info_min_time=1413689820.000, info_max_time=1417322220.000, info_search_time=1417322223.036, high="42.55996", low="2.64004", pred="22.600000"

As before, timestartpos and timeendpos are 0 and 25 for all events. Trying to whittle this down to a minimal test case, but it's looking like there's some dependency on search time.

Re: Summary index event timestamp issue

mlf — Mon, 28 Sep 2020 18:18:15 GMT

I found it. Looks like I've been bumping up against the default MAX_DAYS_HENCE=2. Any events generated with times greater than 2 days past the search time get the last timestamp available in that 2 day window. Worked around by adjusting the search schedule slightly to stay under the 2 day window. Will add a props.conf entry for [stash] at a later date.