splunk summary index

[uagraw01]

In the below screenshot, we can see that from November 6th onwards, there are three sources generated in Splunk; it shows only one "File Collector: DepTrayCaseQty." Splunk created unnecessary two other sources. Because of the creation of two other sources, unwanted duplicate events were also generated. "D:\Splunk\var\spool\splunk\adb0f8d721bf93e3_events.stash_new" and "D:\Splunk\var\spool\splunk\d0d3783e41cf130c_events.stash_new" . Please guide us on how I can fix this issue.

My assumption : Is collect command is not working fine? How to prevent both of those sources from being ingested into Splunk ?

[ITWhisperer]

"from November 6th onwards" begs the question, what changed in your environment on 6th November?

[uagraw01]

@ITWhisperer Number of events are thripled. As well as duplicated data ingested in splunk

[ITWhisperer]

I assume by that you mean there are two extra reports adding to the summary index? So, what else in your environment changed (which may have impacted the summary index)?

[uagraw01]

@ITWhisperer No only one report Triggering the events

[ITWhisperer]

When did the report change? What does search does the current report use? What search did the report use prior to 6th November?

[uagraw01]

@ITWhisperer Can I check those details in _audit index ?

[ITWhisperer]

It is certainly worth looking

[uagraw01]

@ITWhisperer Let me understand correctly, if more than one source is generating that means, more than one summary index ? Multiple source “/var/spool*”  file generation on the same time frame means ?

[ITWhisperer]

summary is the default index for summaries but you can collect to different indexes. I can't tell from your screenshot whether these are for the same index or not.

Perhaps you should collect additional information about these sources e.g. exactly when did they update, what other fields are in the summary events, etc.

[uagraw01]

@ITWhisperer 

Below are screenshot in which you can see from 6th of November we are receiving 3 sources. and before that the source was only one. 

[ITWhisperer]

Rather than pasting pictures, please paste 3 "duplicated" raw events into a code block </>

[uagraw01]

11/06/2023 23:57:02 +1100, info_min_time=1699189200.000, info_max_time=1700571600.000, info_search_time=1700625838.094, foo=3, Mixed=0, CaseQty=64, OrderId=52128969634, TrayQty=35, Location="DEP/AutoDep03", Dimension=2, TrayError=3, OrientationError=1, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4
OrderId = 52128969634host = MSRDC-BPIsource = D:\Splunk\var\spool\splunk\d0d3783e41cf130c_events.stash_newsourcetype = stash

=====================================================================

11/06/2023 23:57:02 +1100, search_name="File Collector: DepTrayCaseQty", search_now=1699279200.000, info_min_time=1699189200.000, info_max_time=1699275600.000, info_search_time=1699279202.226, foo=2, Mixed=0, CaseQty=29, OrderId=52128969634, TrayQty=17, Location="DEP/AutoDep03", Dimension=2, TrayError=2, OrientationError=0, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4
OrderId = 52128969634host = MSRDC-BPIsource = File Collector: DepTrayCaseQtysourcetype = stash

=================================================================

11/06/2023 23:57:02 +1100, info_min_time=1699189200.000, info_max_time=1700398800.000, info_search_time=1700618994.511, foo=3, Mixed=0, CaseQty=64, OrderId=52128969634, TrayQty=35, Location="DEP/AutoDep03", Dimension=2, TrayError=3, OrientationError=1, ProtrusionError=0, CaseTypeId=6210, WidthError=2, reporttype=DepTrayCaseQty, OffCentreError=0, HeightError=0, LengthError=0, PalletLayers=4
OrderId = 52128969634host = MSRDC-BPIsource = D:\Splunk\var\spool\splunk\adb0f8d721bf93e3_events.stash_newsourcetype = stash

[ITWhisperer]

Looking at the info times show that the events were added by different searches

These appear to been executed on 22nd, with different time spans, 5th - 19th and 5th - 21st. These are the searches which have duplicated your events.

I did a BSides presentation a year or so ago about making summary index reports idempotent to avoid duplicate entries. Summary Index Idempotency - Chris Kaye - YouTube

[uagraw01]

@ITWhisperer Thanks, for sharing that valuable video. I have question, consider my below search which I am using to append the result in summary index. But here I am not using any subsearches, so where I can use your suggested workaround here ?

index=ABC (sourcetype=DepTsuEventTrackingUpdate DepTsuEventTrackingUpdate.LocationQualifiedName=Tray* DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!=null AND DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!="TsuUnknownContent") OR (sourcetype=DepTsuEventContentMove) 
| foreach *.OrderId 
    [| eval OrderId=coalesce('OrderId','<<FIELD>>')] 
| replace ProtrusionFront with Protrusion , ProtrusionBack with Protrusion , ProtrusionLeft with Protrusion , ProtrusionRight with Protrusion , ProtrusionTop with Protrusion 
| rename DepTsuEventTrackingUpdate.TsuSuspect.CheckResult.CheckType as Error DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason as TsuSuspectReason DepTsuEventContentMove.SenderFmInstanceName as Location DepTsuEventTrackingUpdate.TsuId as TsuId DepTsuEventContentMove.TsuContent.Quantity as Quantity DepTsuEventContentMove.LocationQualifiedName as TrayLoad DepTsuEventContentMove.TsuContent.CaseTypeId as CaseTypeId
| eval OrientationError=if(Error="Orientation","1","0") , ProtrusionError=if(Error="Protrusion","1","0") , LengthError=if(Error="Length","1","0") , WidthError=if(Error="Width","1","0") , HeightError=if(Error="Height","1","0") , OffCentreError=if(Error="OffCentre","1","0") 
| eval DimensionError=if(LengthError>0 OR WidthError>0 OR HeightError>0, "1","0") 
| eval ErrorQty=(OrientationError+ProtrusionError+DimensionError+OffCentreError) , TrayError=(OrientationError+ProtrusionError+LengthError+WidthError+HeightError+OffCentreError) , TrayError=if(TrayError>0,"1",null) 
| eval Dimension=if(DimensionError>0 AND ErrorQty="1" ,"1","0") , Orientation=if(OrientationError="1" AND ErrorQty="1","1","0") , Protrusion=if(ProtrusionError="1" AND ErrorQty="1","1","0") , Length=if(LengthError="1" AND ErrorQty="1","1","0") , Width=if(WidthError="1" AND ErrorQty="1","1","0") , Height=if(HeightError="1" AND ErrorQty="1","1","0") , OffCentre=if(OffCentreError="1" AND ErrorQty="1","1","0") , Mixed=if(Dimension="0" AND ErrorQty>1,"1","0") 
| eval Layer=if(TrayLoad="PalletInPosition","1",null) , CaseQty=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2",Quantity,null) , Tray=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2","1",null) 
| stats min(_time) as _time values(Location) as Location sum(Layer) as PalletLayers sum(Tray) as TrayQty sum(CaseQty) as CaseQty sum(TrayError) as TrayError sum(Orientation) as OrientationError sum(Length) as LengthError sum(Width) as WidthError sum(Height) as HeightError sum(Protrusion) as ProtrusionError sum(OffCentre) as OffCentreError sum(Dimension) as Dimension sum(Mixed) as Mixed values(CaseTypeId) as CaseTypeId by OrderId 
| eval reporttype="DepTrayCaseQty" 
| eval foo=Dimension+Mixed+OrientationError+ProtrusionError+OffCentreError 
| table _time reporttype OrderId CaseTypeId Location PalletLayers TrayQty CaseQty TrayError foo Dimension Mixed OrientationError LengthError WidthError HeightError ProtrusionError OffCentreError 
| where isnotnull(CaseQty)
| collect index=analyst

[ITWhisperer · Accepted solution]

Assuming _time and OrderId uniquely identify events in the search and summary index, try something like this

index=ABC (sourcetype=DepTsuEventTrackingUpdate DepTsuEventTrackingUpdate.LocationQualifiedName=Tray* DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!=null AND DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason!="TsuUnknownContent") OR (sourcetype=DepTsuEventContentMove) 
| foreach *.OrderId 
    [| eval OrderId=coalesce('OrderId','<<FIELD>>')] 
| replace ProtrusionFront with Protrusion , ProtrusionBack with Protrusion , ProtrusionLeft with Protrusion , ProtrusionRight with Protrusion , ProtrusionTop with Protrusion 
| rename DepTsuEventTrackingUpdate.TsuSuspect.CheckResult.CheckType as Error DepTsuEventTrackingUpdate.TsuSuspect.TsuSuspectReason as TsuSuspectReason DepTsuEventContentMove.SenderFmInstanceName as Location DepTsuEventTrackingUpdate.TsuId as TsuId DepTsuEventContentMove.TsuContent.Quantity as Quantity DepTsuEventContentMove.LocationQualifiedName as TrayLoad DepTsuEventContentMove.TsuContent.CaseTypeId as CaseTypeId
| eval OrientationError=if(Error="Orientation","1","0") , ProtrusionError=if(Error="Protrusion","1","0") , LengthError=if(Error="Length","1","0") , WidthError=if(Error="Width","1","0") , HeightError=if(Error="Height","1","0") , OffCentreError=if(Error="OffCentre","1","0") 
| eval DimensionError=if(LengthError>0 OR WidthError>0 OR HeightError>0, "1","0") 
| eval ErrorQty=(OrientationError+ProtrusionError+DimensionError+OffCentreError) , TrayError=(OrientationError+ProtrusionError+LengthError+WidthError+HeightError+OffCentreError) , TrayError=if(TrayError>0,"1",null) 
| eval Dimension=if(DimensionError>0 AND ErrorQty="1" ,"1","0") , Orientation=if(OrientationError="1" AND ErrorQty="1","1","0") , Protrusion=if(ProtrusionError="1" AND ErrorQty="1","1","0") , Length=if(LengthError="1" AND ErrorQty="1","1","0") , Width=if(WidthError="1" AND ErrorQty="1","1","0") , Height=if(HeightError="1" AND ErrorQty="1","1","0") , OffCentre=if(OffCentreError="1" AND ErrorQty="1","1","0") , Mixed=if(Dimension="0" AND ErrorQty>1,"1","0") 
| eval Layer=if(TrayLoad="PalletInPosition","1",null) , CaseQty=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2",Quantity,null) , Tray=if(TrayLoad="TrayLoad1" OR TrayLoad="TrayLoad2","1",null) 
| stats min(_time) as _time values(Location) as Location sum(Layer) as PalletLayers sum(Tray) as TrayQty sum(CaseQty) as CaseQty sum(TrayError) as TrayError sum(Orientation) as OrientationError sum(Length) as LengthError sum(Width) as WidthError sum(Height) as HeightError sum(Protrusion) as ProtrusionError sum(OffCentre) as OffCentreError sum(Dimension) as Dimension sum(Mixed) as Mixed values(CaseTypeId) as CaseTypeId by OrderId 
| eval reporttype="DepTrayCaseQty" 
| eval foo=Dimension+Mixed+OrientationError+ProtrusionError+OffCentreError 
| table _time reporttype OrderId CaseTypeId Location PalletLayers TrayQty CaseQty TrayError foo Dimension Mixed OrientationError LengthError WidthError HeightError ProtrusionError OffCentreError 
| where isnotnull(CaseQty)
| eval flag=1
| append [search index=analyst
  | eval flag=2]
| eventstats sum(flag) as flags by _time OrderId
| where flags = 1
| fields - flag flags
| collect index=analyst

[uagraw01]

@ITWhisperer I will try this by tommorow when I am on my machine as a workaround.

I am still not figuring out from where two extra stash file created. Please help me to identify those things. What do I need to check? I have checked audit index logs and internal index logs but nothing I have found.

[ITWhisperer]

As I said before, these searches appear to have been executed on 22nd, you should check your audit around these times (for my time zones, this appears to be just before 02:10am and 04:04am)