Knowledge Management

how to retain _time in summary index

AnilPujar
Path Finder

when I try simple below query its taking the current system time instead of _time of original event.

splunk version: 6.6.3

index=indexname | collect index=si

I want the events in the summary index to retain the _time as it is in the primary index. But it's storing the current system time.

0 Karma

hnorvik
Explorer

Looking through all the comments, it is obvious that some don't understand there may be other objectives of moving events from one index to another than creating a plain summary.

There are two objectives that I clearly see here - and there could be more:

  • You want to create a summary of events so it is easier to search through a subset of data (hence summary index)
  • You want to preserve certain _raw events as is before freezing / deleting old buckets (example configuration audit changes)

I have mainly looked into the last item on this list - preserving configuration audit logs for a Cisco ASA beyond the freeze/delete point of 1 year I have set on my index of the original events.

The collect command will just ourput the _raw record to the stash file, it is then useful to get the timestamps correctly added to each.  I used this search:


host=1.2.3.4 message_id=111008
| eval eventtime=strftime(_time,"%Y-%m-%dT%H:%M:%S")
| eval _raw=eventtime + ": " + _raw | collect index=config_change

You may add host= and sourcetype= to the collect command to preserve it, but it will then be counted in  your license. 

Regards H.

0 Karma

bogdan_nicolesc
Communicator

Hi all,

Actually, this is a very good question. The only thing is that is asked wrong ....

How, anyone like myself, could run collect command from an index to another, and retain the original _time of the older index????

Does anyone think of that?

Thank you,
Bogdan.

0 Karma

AnilPujar
Path Finder

Thanks Bogdan and also for your time in trying it out.

Still waiting for answer..

0 Karma

Vijeta
Influencer

@AnilPujar - What is your use case and the collect will give you timestamp of your events in index your are searching on. In case there is no timestamp in your main search only then it takes current system time. You can run in testmode and compare the _time from your main index to the result after collect.

0 Karma

jvishwak
Path Finder

Try this:
index=indexname | eval _time = Timestamp field in event> | collect index=si

0 Karma

AnilPujar
Path Finder

What is the _time precedence for collect command?

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You're not understanding how a summary index works. You should stay away from this until you get more experience

0 Karma

AnilPujar
Path Finder

In Splunk, for everything there is a precedence from conf files to _time

This is how splunk assign _time to the events when getting data in.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Data/HowSplunkextractstimestamps

so similarly due to some precedence i'm missing _time in summary index.

Might be my existing index data unable to recognize timestamp

@skoelpin "So if you don't know the answers stay away from giving answers and I can see your experience with answers made."

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I'm not sure how to interpret this.. If you're unsure that your populating search may not be extracting _time correctly then you have much deeper problems than writing new data to a summary index. Your original search is just piping raw data from one index to another, completely subverting the purpose of a summary index.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I think you're missing the point of a summary index. In your example, you are pushing raw data to the summary index rather than summarized data. You should have a transformational command prior to your collect command. Once the data is transformed, it creates metrics which can be shipped to a summary index. If you use a timechart command then _time will be passed to the summary index

0 Karma

AnilPujar
Path Finder

Can we move this answer as comment to the question, since it is not the answer to the question asked.

index=indexname | collect index=si

What is the _time precedence if I run this command. The same _time of index=indexname should retain?

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Why would I move this as a comment? From a technical standpoint, you are wasting resources and time shipping raw data to a summary index and getting zero benefit of the acceleration a summary index provides. This answer needs visibility for future Splunkers who may be looking to do the same thing

To be clear, DO NOT DO THIS

0 Karma

mayurr98
Super Champion

Well does your events in primary index has timestamp? According to docs, If you use the collect command with a time range of All time and the events do not have timestamps, the current system time is used for the timestamps.

0 Karma

AnilPujar
Path Finder

Hi Mayur,

The data was collected using servicenow addon and i'm not seeing any timestamp being captured additionally.

Only _time mapped with sys_updated_on

0 Karma

nikita_p
Contributor

Hi,
You can add _time in your base search before collecting it in a new index

0 Karma

vnravikumar
Champion
0 Karma

skoelpin
SplunkTrust
SplunkTrust

This solution is useless if he's sending raw data to a summary index..

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...