topic Re: splunk summary index in Splunk Search

splunk summary index

uagraw01 — Mon, 27 Nov 2023 12:31:39 GMT

In the below screenshot, we can see that from November 6th onwards, there are three sources generated in Splunk; it shows only one "File Collector: DepTrayCaseQty." Splunk created unnecessary two other sources. Because of the creation of two other sources, unwanted duplicate events were also generated. "D:\Splunk\var\spool\splunk\adb0f8d721bf93e3_events.stash_new" and "D:\Splunk\var\spool\splunk\d0d3783e41cf130c_events.stash_new" . Please guide us on how I can fix this issue.

My assumption : Is collect command is not working fine? How to prevent both of those sources from being ingested into Splunk ?

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 13:17:46 GMT

"from November 6th onwards" begs the question, what changed in your environment on 6th November?

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 13:25:10 GMT

@ITWhisperer Number of events are thripled. As well as duplicated data ingested in splunk

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 13:24:45 GMT

I assume by that you mean there are two extra reports adding to the summary index? So, what else in your environment changed (which may have impacted the summary index)?

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 14:10:29 GMT

@ITWhisperer No only one report Triggering the events

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 14:12:32 GMT

When did the report change? What does search does the current report use? What search did the report use prior to 6th November?

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 14:17:38 GMT

@ITWhisperer Can I check those details in _audit index ?

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 14:38:03 GMT

It is certainly worth looking

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 15:04:02 GMT

@ITWhisperer Let me understand correctly, if more than one source is generating that means, more than one summary index ? Multiple source “/var/spool*” file generation on the same time frame means ?

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 15:40:58 GMT

summary is the default index for summaries but you can collect to different indexes. I can't tell from your screenshot whether these are for the same index or not.

Perhaps you should collect additional information about these sources e.g. exactly when did they update, what other fields are in the summary events, etc.

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 16:19:18 GMT

@ITWhisperer

Below are screenshot in which you can see from 6th of November we are receiving 3 sources. and before that the source was only one.

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 16:34:41 GMT

Rather than pasting pictures, please paste 3 "duplicated" raw events into a code block </>

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 16:38:48 GMT

11/06/2023 23:57:02 +1100, info_min_time=1699189200.000, info_max_time=1700571600.000, info_search_time=1700625838.094, foo=3, Mixed=0, CaseQty=64, OrderId=52128969634, TrayQty=35, Location="DEP/AutoDep03", Dimension=2, TrayError=3, OrientationError=1, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4 OrderId = 52128969634host = MSRDC-BPIsource = D:\Splunk\var\spool\splunk\d0d3783e41cf130c_events.stash_newsourcetype = stash ===================================================================== 11/06/2023 23:57:02 +1100, search_name="File Collector: DepTrayCaseQty", search_now=1699279200.000, info_min_time=1699189200.000, info_max_time=1699275600.000, info_search_time=1699279202.226, foo=2, Mixed=0, CaseQty=29, OrderId=52128969634, TrayQty=17, Location="DEP/AutoDep03", Dimension=2, TrayError=2, OrientationError=0, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4 OrderId = 52128969634host = MSRDC-BPIsource = File Collector: DepTrayCaseQtysourcetype = stash ================================================================= 11/06/2023 23:57:02 +1100, info_min_time=1699189200.000, info_max_time=1700398800.000, info_search_time=1700618994.511, foo=3, Mixed=0, CaseQty=64, OrderId=52128969634, TrayQty=35, Location="DEP/AutoDep03", Dimension=2, TrayError=3, OrientationError=1, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4 OrderId = 52128969634host = MSRDC-BPIsource = D:\Splunk\var\spool\splunk\adb0f8d721bf93e3_events.stash_newsourcetype = stash

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 17:03:47 GMT

Looking at the info times show that the events were added by different searches

These appear to been executed on 22nd, with different time spans, 5th - 19th and 5th - 21st. These are the searches which have duplicated your events.

I did a BSides presentation a year or so ago about making summary index reports idempotent to avoid duplicate entries. Summary Index Idempotency - Chris Kaye - YouTube

Re: splunk summary index

uagraw01 — Mon, 27 Nov 2023 17:40:47 GMT

@ITWhisperer Thanks, for sharing that valuable video. I have question, consider my below search which I am using to append the result in summary index. But here I am not using any subsearches, so where I can use your suggested workaround here ?

index=ABC (sourcetype=DepTsuEventTrackingUpdate DepTsuEventTrackingUpdate.LocationQualifiedName=Tray* DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!=null AND DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!="TsuUnknownContent") OR (sourcetype=DepTsuEventContentMove) | foreach *.OrderId [| eval OrderId=coalesce('OrderId','<<FIELD>>')] | replace ProtrusionFront with Protrusion , ProtrusionBack with Protrusion , ProtrusionLeft with Protrusion , ProtrusionRight with Protrusion , ProtrusionTop with Protrusion | rename DepTsuEventTrackingUpdate.TsuSuspect.CheckResult.CheckType as Error DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason as TsuSuspectReason DepTsuEventContentMove.SenderFmInstanceName as Location DepTsuEventTrackingUpdate.TsuId as TsuId DepTsuEventContentMove.TsuContent.Quantity as Quantity DepTsuEventContentMove.LocationQualifiedName as TrayLoad DepTsuEventContentMove.TsuContent.CaseTypeId as CaseTypeId | eval OrientationError=if(Error="Orientation","1","0") , ProtrusionError=if(Error="Protrusion","1","0") , LengthError=if(Error="Length","1","0") , WidthError=if(Error="Width","1","0") , HeightError=if(Error="Height","1","0") , OffCentreError=if(Error="OffCentre","1","0") | eval DimensionError=if(LengthError>0 OR WidthError>0 OR HeightError>0, "1","0") | eval ErrorQty=(OrientationError+ProtrusionError+DimensionError+OffCentreError) , TrayError=(OrientationError+ProtrusionError+LengthError+WidthError+HeightError+OffCentreError) , TrayError=if(TrayError>0,"1",null) | eval Dimension=if(DimensionError>0 AND ErrorQty="1" ,"1","0") , Orientation=if(OrientationError="1" AND ErrorQty="1","1","0") , Protrusion=if(ProtrusionError="1" AND ErrorQty="1","1","0") , Length=if(LengthError="1" AND ErrorQty="1","1","0") , Width=if(WidthError="1" AND ErrorQty="1","1","0") , Height=if(HeightError="1" AND ErrorQty="1","1","0") , OffCentre=if(OffCentreError="1" AND ErrorQty="1","1","0") , Mixed=if(Dimension="0" AND ErrorQty>1,"1","0") | eval Layer=if(TrayLoad="PalletInPosition","1",null) , CaseQty=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2",Quantity,null) , Tray=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2","1",null) | stats min(_time) as _time values(Location) as Location sum(Layer) as PalletLayers sum(Tray) as TrayQty sum(CaseQty) as CaseQty sum(TrayError) as TrayError sum(Orientation) as OrientationError sum(Length) as LengthError sum(Width) as WidthError sum(Height) as HeightError sum(Protrusion) as ProtrusionError sum(OffCentre) as OffCentreError sum(Dimension) as Dimension sum(Mixed) as Mixed values(CaseTypeId) as CaseTypeId by OrderId | eval reporttype="DepTrayCaseQty" | eval foo=Dimension+Mixed+OrientationError+ProtrusionError+OffCentreError | table _time reporttype OrderId CaseTypeId Location PalletLayers TrayQty CaseQty TrayError foo Dimension Mixed OrientationError LengthError WidthError HeightError ProtrusionError OffCentreError | where isnotnull(CaseQty) | collect index=analyst

Re: splunk summary index

ITWhisperer — Mon, 27 Nov 2023 18:19:06 GMT

Assuming _time and OrderId uniquely identify events in the search and summary index, try something like this

Re: splunk summary index

uagraw01 — Tue, 28 Nov 2023 16:38:44 GMT

@ITWhisperer I will try this by tommorow when I am on my machine as a workaround.

I am still not figuring out from where two extra stash file created. Please help me to identify those things. What do I need to check? I have checked audit index logs and internal index logs but nothing I have found.

Re: splunk summary index

ITWhisperer — Wed, 29 Nov 2023 10:44:48 GMT

As I said before, these searches appear to have been executed on 22nd, you should check your audit around these times (for my time zones, this appears to be just before 02:10am and 04:04am)