Splunk Search

Why my Saved Search is not writing summary to 'stash' souretype?

zacksoft_wf
Contributor

I am investigating a customer's concern that this  particular search is not writing summary to 'stash' sourcetype. This is the SPL we have. I am relatively new to summary index. Any pointers where I should be looking at ?

| tstats summariesonly=true count sum(Web.bytes_in) AS total_bytes_in sum(Web.bytes_out) AS total_bytes_out from datamodel=Web where sourcetype="qnet:proxysg:access*" groupby _time span=1d Web.category
| rename Web.category AS category
| eval total_bytes=total_bytes_in + total_bytes_out
| collect index=security_summary source="SavedSearch.Qnet_Daily_Category_Stats" sourcetype="SavedSearch.Qnet_Daily_Category_Stats" 
Labels (1)
0 Karma
1 Solution

jamie00171
Communicator

Hi @zacksoft_wf 

in the collect command you are specifying a different sourcetype: SavedSearch.Qnet_Daily_Category_Stats, if you remove that then the default of "stash" will be used.

FYI - any parsing rules etc. you have in props.conf for that sourcetype "SavedSearch.Qnet_Daily_Category_Stats" will no longer be applied to the events before they get to the security_summary index.

Thanks, 

Jamie

View solution in original post

0 Karma

jamie00171
Communicator

Hi @zacksoft_wf 

in the collect command you are specifying a different sourcetype: SavedSearch.Qnet_Daily_Category_Stats, if you remove that then the default of "stash" will be used.

FYI - any parsing rules etc. you have in props.conf for that sourcetype "SavedSearch.Qnet_Daily_Category_Stats" will no longer be applied to the events before they get to the security_summary index.

Thanks, 

Jamie

0 Karma

zacksoft_wf
Contributor

@jamie00171  - Thank Jamie. I am thinking to rerun my query with collect statement and removing the source type from there, so that it will default get saved into 'stash' sourcetype.

My concern is, that particular summary-index already collects summary data from other searches. Will my SPL overwrite any of the existing fields ?  
And after running my current query , how do I know that the data get stored in stash, Will I see the fields (from my SPL) get added into the interesting field side bar, and thats how I know it is now going to stash ?

0 Karma

jamie00171
Communicator

Hi @zacksoft_wf 

When you use the collect command, Splunk writes the search results to $SPLUNK_HOME/var/spool/splunk directory/ directory on the disk of the host that executed the search. By default Splunk is configured (via $SPLUNK_HOME/etc/system/default/inputs.conf) to monitor this directory for any new files and put the contents through the indexing pipeline similar to a standard file input.

The index used in the collect command determines which index the events go to.

The source and sourcetype you use in the collect command will determine what props/transforms (field extractions etc.) apply when indexing (e.g. if the sourcetype creates indexed fields etc.) and the same source and sourcetype will have to be used when searching for the events and therefore the search time parsing rules of the associated source and sourcetype are applied which decide the interesting fields on the left hand side. 

"Will my SPL overwrite any of the existing fields ?" - if you don't change the props / transforms config then nothing will be overwritten in terms of field extractions, it's more likely the filed extractions won't work for the data written to the summary index from the new search won't work, depending on how similar the new data is compared with the data that is written from the existing searches. 

If the data is different you would typically use different source types with config specific to the format of the data to make sure the data gets parsed correctly.

Also, typically the only reason you'd put different types of data into the same index would be if it searched together (i.e. index=myindex sourcetype=mysourcetype OR sourcetype=mysecondsourcetype), if you put a lot of different types of data into the same index it can impact search performance. (I can go into why this is the case in detail if you're interested).

 

Hope this helps, 

 

Jamie

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...