topic Re: How to modify _time when running summary index on a scheduled search? in Splunk Search

How to modify _time when running summary index on a scheduled search?

LearningGuy — Fri, 08 Mar 2024 15:53:35 GMT

Hello,
How to modify _time when running summary index on a scheduled search?
Please suggest. I appreciate your help. Thank you

When running summary index on a scheduled search, by default, _time was set to info_min_time, (start time of a search duration), instead of search_now (time when the search run)
So, if at this current time I collect the summary index in the last 30 day , the _time will be set to the last 30 days , instead of current time.

The problem is if I run a search in the past 24 hours, the data won't show up because the _time is dated the last 30 days, so I had to search in the past 30 days

Re: How to modify _time when running summary index on a scheduled search?

marnall — Fri, 08 Mar 2024 21:46:04 GMT

You could manually set the _time value to the time when the scheduled search is run.

e.g.

| eval _time = now()

Re: How to modify _time when running summary index on a scheduled search?

LearningGuy — Fri, 08 Mar 2024 21:47:55 GMT

Hello,

Have you tested it yourself?
I have tried your suggestion, but it did not work.
By default, _time was set to info_min_time

Thanks,

Marius

Re: How to modify _time when running summary index on a scheduled search?

marnall — Sat, 09 Mar 2024 13:02:38 GMT

Yes, I have a summary search which ends in:

| eval _time = _time + 3600

This set the timestamp of the summary-indexed events to one hour in the future. Then, when I search for the summarized events using the time filter of "Last 24 hours", it does not find any events (as expected). When I search for the summarized events with a custom time filter from +1m to +2h, then it does find the events, timestamped one hour in the future.

Thus this method should be useful for setting the timestamps of your summarized index events to be in your expected search window.

Re: How to modify _time when running summary index on a scheduled search?

LearningGuy — Mon, 11 Mar 2024 14:44:06 GMT

Hi @marnall

I tried your suggestion, but the _time always set to info_min_time

My search time is this morning: Mar 11 2024 10:12:00 EDT
Time Frame is: last 30 day, so
info_min_time (start time: Feb 10 2024 00:00:00 EST)
info_max_time (end time: Mar 11 2024 00:00:00 EDT)

_time is set to info_min_time as seen below

Can you set your search time to to last 30 day and run the collect command with testmode=true and share your results?

| collect index= summary testmode=true addtime=true file=summary_test_1.stash_new name="summary_test_1" marker="report=\"summary_test_1\""

You should have _raw field that contains all the fields, including _time before getting pushed by collect command. See my below output. Please share yours.. Thanks

Re: How to modify _time when running summary index on a scheduled search?

marnall — Tue, 12 Mar 2024 20:53:00 GMT

Sure thing. For testing I am using this SPL: (time range set to "Last 30 Days")

It produces the following output:

_time	sourcetype	_raw	othertestfield
2024-03-12T22:50:05.000+01:00	splunkd	03/12/2024 22:50:05 +0100, info_min_time=1707606000.000, info_max_time=1710276605.000, info_search_time=1710276605.390, othertestfield=test1, orig_sourcetype=splunkd	test1
2024-03-12T22:50:05.000+01:00	splunkd_access	03/12/2024 22:50:05 +0100, info_min_time=1707606000.000, info_max_time=1710276605.000, info_search_time=1710276605.390, othertestfield=test1, orig_sourcetype=splunkd_access	test1
2024-03-12T22:50:05.000+01:00	splunkd_access	03/12/2024 22:50:05 +0100, info_min_time=1707606000.000, info_max_time=1710276605.000, info_search_time=1710276605.390, othertestfield=test1, orig_sourcetype=splunkd_access	test1
2024-03-12T22:50:05.000+01:00	splunkd_access	03/12/2024 22:50:05 +0100, info_min_time=1707606000.000, info_max_time=1710276605.000, info_search_time=1710276605.390, othertestfield=test1, orig_sourcetype=splunkd_access	test1
2024-03-12T22:50:05.000+01:00	splunkd_access	03/12/2024 22:50:05 +0100, info_min_time=1707606000.000, info_max_time=1710276605.000, info_search_time=1710276605.390, othertestfield=test1, orig_sourcetype=splunkd_access	test1

I ran the search at 21:50 CET, and the _time field shows the current time plus 3600 seconds.

Re: How to modify _time when running summary index on a scheduled search?

LearningGuy — Wed, 13 Mar 2024 03:42:16 GMT

Hi @marnall

Your suggestion worked fine. I accepted this as a solution. Thank you so much for your help.
It looks like the reason it didn't work earlier because I assigned
eval _time = info_max_time, but I didn't put "addinfo", so it went to default value which is info_min_time.

Can you test on your end if _time set to info_min_time, if you don't use/remove the following eval?
Thanks

| eval _time = now() + 3600

Re: How to modify _time when running summary index on a scheduled search?

marnall — Wed, 13 Mar 2024 20:29:03 GMT

Sure. When I remove that line, I get:

index=_internal | table _time sourcetype | head 5 | eval othertestfield="test2" | collect index=summary testmode=true addtime=true

When setting "Last 30 days" with the time picker, it produces another 5 rows, but I only paste the first one for brevity:

_time	sourcetype	_raw	othertestfield
2024-03-13T21:13:38.999+01:00	splunkd	03/13/2024 21:13:38 +0100, info_min_time=1707692400.000, info_max_time=1710361482.000, info_search_time=1710361482.294, othertestfield=test2, orig_sourcetype=splunkd	test2

It seems the _time field is not set automatically to info_min_time, or else it should show something like 02/12/2024 in the _time part of the _raw field

Re: How to modify _time when running summary index on a scheduled search?

LearningGuy — Thu, 14 Mar 2024 01:48:39 GMT

Hi @marnall

Thank you again for your help
I did some test, it appears that if we include _time in | table, _time within _raw will follow the _time field, but if we use | table without _time, the _time will be set to info_min_time
This is also the reason why it didn't work when I tried your first suggestion.

Can you try the following and let me know the result?
Thank you so much

index=_internal | table sourcetype | head 5 | eval othertestfield="test2" | collect index=summary testmode=true addtime=true

Re: How to modify _time when running summary index on a scheduled search?

marnall — Fri, 15 Mar 2024 21:17:06 GMT

Sure thing. When running that search over "Last 30 days", in the resulting _raw field I get:

02/14/2024 00:00:00 +0100, info_min_time=1707865200.000, info_max_time=1710537355.000, info_search_time=1710537355.622, othertestfield=test2, orig_sourcetype=splunkd

which does indeed look like the _time value has defaulted to info_min_time