Knowledge Management

How can I preserver original fields in a summary index?

joydeep741
Path Finder

When I summary index, my data's original fields are lost.
How can I preserve original fields in a Summary Index?

Tags (2)
0 Karma

woodcock
Esteemed Legend

That is how it is supposed to work. The summary part means just that. It only saves the fields that are present in your final results and even then, some, like host are overridden and moved to orig_* values like orig_host. If you need the fields then you must preserver then through to the end of your search results.

0 Karma

wrighke
Explorer

One option is to prepend the original fields before sending to the summary index...

| foreach _* *
    [| eval orig_<<MATCHSTR>>=<<FIELD>>]

(You may want to only prepend the fields you actually need, and which get overwritten, e.g. host, source, sourcetype, instead of _* *)

0 Karma

ngatchasandra
Builder

Hi joy,

This must be due to the fact that your field extractions are applied to the sourcetype given to the _raw indexed data . I think if you try to run your query in the search app using | collect index="summaryIndex", the fields are there.

To fix this, try to recreate your field extraction regex's to also be applied to a sourcetype of ''application''.

0 Karma

joydeep741
Path Finder
  1. “Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

  1. INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi joy

The stats command above would only have two fields: Requests and host_sourcetype . These are the only fields created by that search. Which other fields are you expecting to see?

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi joydeep41
Can you share what search you are running where you can't see the original fields? I summary search would typically end with a summarizing command like stats or timechart and not raw data. The meta data fields (source, sourcetype etc.) should get the "original_" prefix added to them.

j

0 Karma

joydeep741
Path Finder
  1. “Original Search Query” : Before summary indexing Query searches raw data of last one day (yesterday’s data) :

index=dotcom_odin
sourcetype="odin_ws_access"

| eval host_sourcetype= host+"_"+sourcetype
| timechart span=5m count as "Requests" by host_sourcetype limit=100

  1. INDEX POPULATING SEARCH :-

index=dotcom_odin
sourcetype="odin_ws_access"
| eval host_sourcetype= host+"_"+sourcetype
| sitimechart span=5m count as "Requests" by host_sourcetype limit=100

After the summary index is complete I see all my original fields lost. I want to use them in queries. Is there a way to preserve them in summary index ?

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...