Knowledge Management

How do timestamps work with summary indexing?

christoffertoft
Communicator

Im just now learning about summary indexing and have set up a search to run every hour, putting the results in a specific summary index. When I run the saved search in the splunk search bar, I get the unique timestamps for each event, however, if I search the index using the following search string:

index=mysummaryindex report=myreport 

The timestamps are all from 8:00:00.000 AM , 9:00:00.000 AM, etc. It seems like the timestamp is from the time when the saved search is ran. Is this expected behaviour? I was hoping to be able to see the original timestamps like when I run the search manually?

Regards, and thanks.

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

I marked your code as code, but it looks like some of the rex code is still missing. If you are wanting to collect each of the messages as individual messages, then you need to move toward the collect command, probably with a table in front of it to reduce the amount of data. On the other hand, if you want summary data -- say, a count of occurrences of each Message with the first and last _time in each hour -- then you need to build the data that you want to keep into the search before sending it to the summary index (also probably with collect, given your leanings).

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I marked your code as code, but it looks like some of the rex code is still missing. If you are wanting to collect each of the messages as individual messages, then you need to move toward the collect command, probably with a table in front of it to reduce the amount of data. On the other hand, if you want summary data -- say, a count of occurrences of each Message with the first and last _time in each hour -- then you need to build the data that you want to keep into the search before sending it to the summary index (also probably with collect, given your leanings).

0 Karma

christoffertoft
Communicator

@DalJeanis thanks. the rex wasn't really important for the question but I thank you for the fix 🙂

I think we'll be moving towards Splunk ES and accelerated data models for the project instead of using summary indexing

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Since it is a summary table, what timestamp do you want? The one from the first event summarized, the average of the times, the median of the times, or that last event's timestamp? The reason for the timestamp as it is is to show the time of the beginning of the data that could be in the data. That is so the data can be searched to be within the timeframe the data is supposed to be summarizing. If you want a specific timestamp for the summary data, put a field in the summary data that is a timestamp that you want it to have. That way you can look at a timestamp for the summarized data the way you want it to be. Otherwise the timestamp of the summary event will be the beginning of the timeframe for the data within the summary event.

christoffertoft
Communicator

I want the time stamp when it was first summarized. I want the data to look "exactly" like as if it was gathered from the original index.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Are you summarizing each and every event as another event? If so, look at the collect command documentation where it tells you how to copy data to another index (usually a summary index) instead of using the summary indexing scheduled search.

If not, then I fail to understand how you can summarize data and have it look exactly like the original.

adonio
Ultra Champion

@christoffertoft,
how theresults from your search looks like?
will recommend to follow @cpetterborg last comment and use the | collect command for more summary indexing options

0 Karma

jackreeves
Explorer

I am running a Summary Index based on a lookup table. However I want to change the summary index timestamp to "Month/Year" field (see below).

| inputlookup inventory.csv
| stats values(count) as count by Month/Year Month FamilyCod ProductCod

Month/Year field is in following format "%Y/%m". Could anyone advise?

0 Karma

adonio
Ultra Champion

can you share the search you are running?
are you using the | collect command ? or did you enable summary indexes from GUI?
read here in detail all the way through to learn more (3 long pages):
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Usesummaryindexing?r=searchtip

0 Karma

christoffertoft
Communicator

@adonio Any suggestions?

0 Karma

christoffertoft
Communicator

the search im running is (masked):

 starthoursago=2 endhoursago=1 [|inputlookup servers | search host=*nt* | eval host=host+"*"] source=mySource rex field=Message ""

the summary-index check is enabled in the GUI (Settings -> Searches -> the search).

The added field is report=myreport.

When I try to extract the information in a search (to be used in a dashboard) i do index=myindex report=myreport. It shows the wrong timestamps.

0 Karma

alemarzu
Motivator
0 Karma

christoffertoft
Communicator

Like I said, the _time values exist in the original search.. If i do |eval _time=now() All the values will be preformatted to indexing time and not the original timestamp..

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...