Looking for guidance for summary indexing usecase for splunk environment.
Our splunk environment is shared between 2 departments A and B. I belong to B.
splunk environment has
1 search head
5 indexers [ 4 for Department A and 1 for Department B]
We have an existing report built by previous employee which goes to business which looks at data starting 2016, this report was built using subsearches, due to its limitations the report has been sending inaccurate results as the data has been growing.
I wanted to utilize summary indexing to redo this report.
What i have done so far
1. Created custom index say "aplha_summry" on department B indexer [other 4 indexers do not have this indexes due to data separation compliance]
2. Created scheduled search which runs once a day on the search head which will populate the data to my newly created summary index "aplha_summry", i enabled summary indexing option on the scheduled search screen and chose my "aplha_summry" index from the dropdown.
I did not have trouble in creating and saving this saved search.
3. My scheduled search runs successfuly [i could see Successfully wrote file to '/abc/apps/splunk/var/spool/splunk/RMD575e764fb0df3e7a0_28226874.stash_new' when i look at inspect job log] however i dont see any data in my summary index "aplha_summry".
I have gone through some of the related question on this topic
My splunk version is 6.5.2.Currently our search head does not store data locally and isn't configured to forward data. I am not sure if they allow our department to have index on the search head.
What are my options to send down the data to my department B indexer "aplha_summry". If this is not possible what is the closest thing that i could do with my current setup.
index="abc" earliest=01/01/2016:0:0:0 latest=@d |chart sum(accesscount) as cnt
You need to create the index (it will go into
indexes.conf) on the Search Head, or, better yet, on the Indexers. See here:
Thanks woodcock for sharing the link, this is useful but doesn't exactly work for my scenario as we only own 1 of the indexer out of 5 , this approach needs the SH to be configured the data in a load balanced way to its all of the search peers[as we cannot send it only to our indexer as other dept data is present as well], in my case the index can only be created on the box that is owned by my department.
Do you know with these constraints is summary indexing an still option for my setup?
For now I have used outputlookup approach to take care of this and it should work for few years but it isn't efficient as every day search looks at data from 2016 unlike summary indexing populating search which looks at 1 day at a time once caught up.
I will upvote it for the guidance.
If only users on your search head need to see the data AND if the data is reasonably small then it should be OK to keep on your Search Head, especially if you are using CSVs now (which most definitely are on your Search Head). All you need to do is create the Index on your Search Head, restart Splunk and you are good-to-go.
Thanks Woodcock for the suggestion. I am having conversations with my team to see if they can allow creation of my index on all 5 indexes and enable data forwarding from SH to all search peers to get this summary indexing working, in the interim my outputlookup approach is doing the task.