topic Re: How can I preserver original fields in a summary index? in Knowledge Management

How can I preserver original fields in a summary index?

joydeep741 — Thu, 28 Jan 2016 12:49:51 GMT

When I summary index, my data's original fields are lost.
How can I preserve original fields in a Summary Index?

Re: How can I preserver original fields in a summary index?

jbjerke_splunk — Thu, 28 Jan 2016 13:02:06 GMT

Hi joydeep41
Can you share what search you are running where you can't see the original fields? I summary search would typically end with a summarizing command like stats or timechart and not raw data. The meta data fields (source, sourcetype etc.) should get the "original_" prefix added to them.

Re: How can I preserver original fields in a summary index?

ngatchasandra — Thu, 28 Jan 2016 13:15:56 GMT

Hi joy,

This must be due to the fact that your field extractions are applied to the sourcetype given to the _raw indexed data . I think if you try to run your query in the search app using | collect index="summaryIndex", the fields are there.

To fix this, try to recreate your field extraction regex's to also be applied to a sourcetype of ''application''.

Re: How can I preserver original fields in a summary index?

joydeep741 — Tue, 29 Sep 2020 08:36:26 GMT

“Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

Re: How can I preserver original fields in a summary index?

joydeep741 — Tue, 29 Sep 2020 08:36:29 GMT

“Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

Re: How can I preserver original fields in a summary index?

jbjerke_splunk — Thu, 28 Jan 2016 14:05:17 GMT

Hi joy

The stats command above would only have two fields: Requests and host_sourcetype . These are the only fields created by that search. Which other fields are you expecting to see?

Re: How can I preserver original fields in a summary index?

wrighke — Tue, 05 Nov 2019 02:13:15 GMT

One option is to prepend the original fields before sending to the summary index...

| foreach _* *
    [| eval orig_<<MATCHSTR>>=<<FIELD>>]

(You may want to only prepend the fields you actually need, and which get overwritten, e.g. host, source, sourcetype, instead of _* *)

Re: How can I preserver original fields in a summary index?

woodcock — Tue, 05 Nov 2019 03:31:02 GMT

That is how it is supposed to work. The summary part means just that. It only saves the fields that are present in your final results and even then, some, like host are overridden and moved to orig_* values like orig_host. If you need the fields then you must preserver then through to the end of your search results.