I have an application that generates a value that I pull the highest value for each day.
Right now the entire app log is in my indexer, but I only need to store the highest app value and the date stamp for that date. I'd like to do this to save space on my indexer since I have limited resources to use.
I initially created a csv based lookup file with a search that pulled the _time and app values I need, but I then realized I cannot properly address the time field in a search that uses the lookup file.
Here is an example of how that data looks:
I need to either configure this lookup file or possibly a KV store in order for me to be able to pull the app values in a search by a given hour period, day, week, month, year to date, current year, previous year, etc. I have access to over two years worth of data in my indexer to build the initial lookup file. I plan on appending the lookup file each data with the new value.
It would be a bonus if I could use the time picker to choose my date ranges. I have tried a few searches with date references in the search but those did not work. I no longer have those failed test searches.
I have IMO spent too much time trying to get this to work on my own, so I am reaching out to the community for help.
have you ever thought of using a Summary Index instead of a lookup?
You could schedule a daily search (for example at 1.00) a daily search obtaining a result every day with all the information you need and then save this result in a Summary Index, consuming a really negligible amount of disk space and having the info that you they serve.
You can do this using the collect command (see https://docs.splunk.com/Documentation/Splunk/7.3.0/SearchReference/Collect).
Then you can have these information searching on this Summary index.