how to retain _time in summary index

[AnilPujar]

when I try simple below query its taking the current system time instead of _time of original event.

splunk version: 6.6.3

index=indexname | collect index=si

I want the events in the summary index to retain the _time as it is in the primary index. But it's storing the current system time.

[hnorvik]

Looking through all the comments, it is obvious that some don't understand there may be other objectives of moving events from one index to another than creating a plain summary.

There are two objectives that I clearly see here - and there could be more:

• You want to create a summary of events so it is easier to search through a subset of data (hence summary index)
• You want to preserve certain _raw events as is before freezing / deleting old buckets (example configuration audit changes)

I have mainly looked into the last item on this list - preserving configuration audit logs for a Cisco ASA beyond the freeze/delete point of 1 year I have set on my index of the original events.

The collect command will just ourput the _raw record to the stash file, it is then useful to get the timestamps correctly added to each.  I used this search:


host=1.2.3.4 message_id=111008
| eval eventtime=strftime(_time,"%Y-%m-%dT%H:%M:%S")
| eval _raw=eventtime + ": " + _raw | collect index=config_change

You may add host= and sourcetype= to the collect command to preserve it, but it will then be counted in  your license. 

Regards H.

[bogdan_nicolesc]

Hi all,

Actually, this is a very good question. The only thing is that is asked wrong ....

How, anyone like myself, could run collect command from an index to another, and retain the original _time of the older index????

Does anyone think of that?

Thank you,
Bogdan.

[AnilPujar]

Thanks Bogdan and also for your time in trying it out.

Still waiting for answer..

[Vijeta]

@AnilPujar - What is your use case and the collect will give you timestamp of your events in index your are searching on. In case there is no timestamp in your main search only then it takes current system time. You can run in testmode and compare the _time from your main index to the result after collect.

[jvishwak]

Try this:
index=indexname | eval _time = Timestamp field in event> | collect index=si

[AnilPujar]

What is the _time precedence for collect command?

[skoelpin]

You're not understanding how a summary index works. You should stay away from this until you get more experience

[AnilPujar]

In Splunk, for everything there is a precedence from conf files to _time

This is how splunk assign _time to the events when getting data in.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Data/HowSplunkextractstimestamps

so similarly due to some precedence i'm missing _time in summary index.

Might be my existing index data unable to recognize timestamp

@skoelpin "So if you don't know the answers stay away from giving answers and I can see your experience with answers made."

[skoelpin]

I'm not sure how to interpret this.. If you're unsure that your populating search may not be extracting _time correctly then you have much deeper problems than writing new data to a summary index. Your original search is just piping raw data from one index to another, completely subverting the purpose of a summary index.

[skoelpin]

I think you're missing the point of a summary index. In your example, you are pushing raw data to the summary index rather than summarized data. You should have a transformational command prior to your collect command. Once the data is transformed, it creates metrics which can be shipped to a summary index. If you use a timechart command then _time will be passed to the summary index

[AnilPujar]

Can we move this answer as comment to the question, since it is not the answer to the question asked.

index=indexname | collect index=si

What is the _time precedence if I run this command. The same _time of index=indexname should retain?

[skoelpin]

Why would I move this as a comment? From a technical standpoint, you are wasting resources and time shipping raw data to a summary index and getting zero benefit of the acceleration a summary index provides. This answer needs visibility for future Splunkers who may be looking to do the same thing

To be clear, DO NOT DO THIS

[mayurr98]

Well does your events in primary index has timestamp? According to docs, If you use the collect command with a time range of All time and the events do not have timestamps, the current system time is used for the timestamps.

[AnilPujar]

Hi Mayur,

The data was collected using servicenow addon and i'm not seeing any timestamp being captured additionally.

Only _time mapped with sys_updated_on

[nikita_p]

Hi,
You can add _time in your base search before collecting it in a new index

[vnravikumar]

Hi @AnilPujar

Please check this solution

https://answers.splunk.com/answers/78756/is-it-possible-to-rewrite-the-time-value-for-summary-index-...

[skoelpin]

This solution is useless if he's sending raw data to a summary index..