Knowledge Management

How can I preserver original fields in a summary index?

joydeep741
Path Finder

When I summary index, my data's original fields are lost.
How can I preserve original fields in a Summary Index?

Tags (2)
0 Karma

woodcock
Esteemed Legend

That is how it is supposed to work. The summary part means just that. It only saves the fields that are present in your final results and even then, some, like host are overridden and moved to orig_* values like orig_host. If you need the fields then you must preserver then through to the end of your search results.

0 Karma

wrighke
Explorer

One option is to prepend the original fields before sending to the summary index...

| foreach _* *
    [| eval orig_<<MATCHSTR>>=<<FIELD>>]

(You may want to only prepend the fields you actually need, and which get overwritten, e.g. host, source, sourcetype, instead of _* *)

0 Karma

ngatchasandra
Builder

Hi joy,

This must be due to the fact that your field extractions are applied to the sourcetype given to the _raw indexed data . I think if you try to run your query in the search app using | collect index="summaryIndex", the fields are there.

To fix this, try to recreate your field extraction regex's to also be applied to a sourcetype of ''application''.

0 Karma

joydeep741
Path Finder
  1. “Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

  1. INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi joy

The stats command above would only have two fields: Requests and host_sourcetype . These are the only fields created by that search. Which other fields are you expecting to see?

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi joydeep41
Can you share what search you are running where you can't see the original fields? I summary search would typically end with a summarizing command like stats or timechart and not raw data. The meta data fields (source, sourcetype etc.) should get the "original_" prefix added to them.

j

0 Karma

joydeep741
Path Finder
  1. “Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

  1. INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...