We have SH clustered environment and we have a lookup which is collected daily and this comes to SH and is present in all the SH cluster members correctly. But we have a requirement to index this (for legal and audit purposes), so the customer can see how the data looked like previously (say 20 days ago). What's the best way to index this csv/lookup file which is present in the SH cluster? when i thinking more and more it becomes complex and complex
(Options I thought so far: if I write an inputs.conf and pull this from a SH, this will have copy in every SH member and the inputs will be replicated too thus causing multiple copies to be indexed. Another alternative I thought is to run a Universal forwarder separately ..hmm..)
Any other easy methods to pull a lookup file from SH to Indexer?
You can just use the summary indexing (saved search fetching data from lookup and saving onto index) to achieve the same. Since, it's a saved search, the SHC will take care of it's execution (once). See this on how to setup a saved search to do summary indexing.
I would run the same search at the same time but remove the outputlookup command and append the collect command so that it gets collected into an index. Be sure you're forwarding events from the SHC to the indexers as per best practices first.
... | collect index=myIndex
We do this using
summary indexing by scheduling a search like this to run daily (or whatever the requirement is):
|inputlookup MyFile.csv | collect index=MySummaryIndex