topic Re: how to retain _time in summary index in Knowledge Management

how to retain _time in summary index

AnilPujar — Wed, 16 Jan 2019 08:09:45 GMT

when I try simple below query its taking the current system time instead of _time of original event.

splunk version: 6.6.3

index=indexname | collect index=si

I want the events in the summary index to retain the _time as it is in the primary index. But it's storing the current system time.

Re: how to retain _time in summary index

vnravikumar — Wed, 16 Jan 2019 08:16:31 GMT

Hi @AnilPujar

Please check this solution

https://answers.splunk.com/answers/78756/is-it-possible-to-rewrite-the-time-value-for-summary-index-events.html

Re: how to retain _time in summary index

nikita_p — Wed, 16 Jan 2019 09:02:10 GMT

Hi,
You can add _time in your base search before collecting it in a new index

Re: how to retain _time in summary index

mayurr98 — Wed, 16 Jan 2019 09:09:58 GMT

Well does your events in primary index has timestamp? According to docs, If you use the collect command with a time range of All time and the events do not have timestamps, the current system time is used for the timestamps.

Re: how to retain _time in summary index

skoelpin — Wed, 16 Jan 2019 13:17:56 GMT

I think you're missing the point of a summary index. In your example, you are pushing raw data to the summary index rather than summarized data. You should have a transformational command prior to your collect command. Once the data is transformed, it creates metrics which can be shipped to a summary index. If you use a timechart command then _time will be passed to the summary index

Re: how to retain _time in summary index

skoelpin — Wed, 16 Jan 2019 13:19:13 GMT

This solution is useless if he's sending raw data to a summary index..

Re: how to retain _time in summary index

AnilPujar — Tue, 05 Feb 2019 05:43:45 GMT

Can we move this answer as comment to the question, since it is not the answer to the question asked.

index=indexname | collect index=si

What is the _time precedence if I run this command. The same _time of index=indexname should retain?

Re: how to retain _time in summary index

jvishwak — Tue, 05 Feb 2019 06:32:54 GMT

Try this:
index=indexname | eval _time = Timestamp field in event> | collect index=si

Re: how to retain _time in summary index

skoelpin — Tue, 05 Feb 2019 15:55:53 GMT

Why would I move this as a comment? From a technical standpoint, you are wasting resources and time shipping raw data to a summary index and getting zero benefit of the acceleration a summary index provides. This answer needs visibility for future Splunkers who may be looking to do the same thing

To be clear, DO NOT DO THIS

Re: how to retain _time in summary index

AnilPujar — Tue, 19 Feb 2019 05:25:26 GMT

What is the _time precedence for collect command?

Re: how to retain _time in summary index

skoelpin — Tue, 19 Feb 2019 14:40:50 GMT

You're not understanding how a summary index works. You should stay away from this until you get more experience

Re: how to retain _time in summary index

Vijeta — Tue, 19 Feb 2019 18:41:56 GMT

@AnilPujar - What is your use case and the collect will give you timestamp of your events in index your are searching on. In case there is no timestamp in your main search only then it takes current system time. You can run in testmode and compare the _time from your main index to the result after collect.

Re: how to retain _time in summary index

bogdan_nicolesc — Wed, 10 Apr 2019 15:44:08 GMT

Hi all,

Actually, this is a very good question. The only thing is that is asked wrong ....

How, anyone like myself, could run collect command from an index to another, and retain the original _time of the older index????

Does anyone think of that?

Thank you,
Bogdan.

Re: how to retain _time in summary index

AnilPujar — Wed, 30 Sep 2020 00:16:47 GMT

Hi Mayur,

The data was collected using servicenow addon and i'm not seeing any timestamp being captured additionally.

Only _time mapped with sys_updated_on

Re: how to retain _time in summary index

AnilPujar — Thu, 25 Apr 2019 09:29:42 GMT

Thanks Bogdan and also for your time in trying it out.

Still waiting for answer..

Re: how to retain _time in summary index

AnilPujar — Thu, 25 Apr 2019 10:43:18 GMT

In Splunk, for everything there is a precedence from conf files to _time

This is how splunk assign _time to the events when getting data in.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Data/HowSplunkextractstimestamps

so similarly due to some precedence i'm missing _time in summary index.

Might be my existing index data unable to recognize timestamp

@skoelpin "So if you don't know the answers stay away from giving answers and I can see your experience with answers made."

Re: how to retain _time in summary index

skoelpin — Thu, 25 Apr 2019 11:17:51 GMT

I'm not sure how to interpret this.. If you're unsure that your populating search may not be extracting _time correctly then you have much deeper problems than writing new data to a summary index. Your original search is just piping raw data from one index to another, completely subverting the purpose of a summary index.

Re: how to retain _time in summary index

hnorvik — Fri, 22 Oct 2021 19:55:59 GMT

Looking through all the comments, it is obvious that some don't understand there may be other objectives of moving events from one index to another than creating a plain summary.

There are two objectives that I clearly see here - and there could be more:

You want to create a summary of events so it is easier to search through a subset of data (hence summary index)
You want to preserve certain _raw events as is before freezing / deleting old buckets (example configuration audit changes)

I have mainly looked into the last item on this list - preserving configuration audit logs for a Cisco ASA beyond the freeze/delete point of 1 year I have set on my index of the original events.

The collect command will just ourput the _raw record to the stash file, it is then useful to get the timestamps correctly added to each. I used this search:

host=1.2.3.4 message_id=111008
| eval eventtime=strftime(_time,"%Y-%m-%dT%H:%M:%S")
| eval _raw=eventtime + ": " + _raw | collect index=config_change

You may add host= and sourcetype= to the collect command to preserve it, but it will then be counted in your license.

Regards H.