topic Creating a lookup file to provide data to a dashboard search with time stamps in Splunk Search

Creating a lookup file to provide data to a dashboard search with time stamps

arrcee — Wed, 30 Sep 2020 01:17:57 GMT

I have an application that generates a value that I pull the highest value for each day.
Right now the entire app log is in my indexer, but I only need to store the highest app value and the date stamp for that date. I'd like to do this to save space on my indexer since I have limited resources to use.
I initially created a csv based lookup file with a search that pulled the _time and app values I need, but I then realized I cannot properly address the time field in a search that uses the lookup file.
Here is an example of how that data looks:
_time app_value
2019-02-26 783
2019-02-27 16091
2019-02-28 11870
2019-03-01 1575
2019-03-02 433
2019-03-03 224
2019-03-04 1893
2019-03-05 10223
2019-03-06 11116
2019-03-07 12822
2019-03-08 1835
2019-03-09 44

I need to either configure this lookup file or possibly a KV store in order for me to be able to pull the app values in a search by a given hour period, day, week, month, year to date, current year, previous year, etc. I have access to over two years worth of data in my indexer to build the initial lookup file. I plan on appending the lookup file each data with the new value.

It would be a bonus if I could use the time picker to choose my date ranges. I have tried a few searches with date references in the search but those did not work. I no longer have those failed test searches.

I have IMO spent too much time trying to get this to work on my own, so I am reaching out to the community for help.
Thank You!!

Re: Creating a lookup file to provide data to a dashboard search with time stamps

gcusello — Fri, 12 Jul 2019 12:39:51 GMT

Hi arrcee,
have you ever thought of using a Summary Index instead of a lookup?
You could schedule a daily search (for example at 1.00) a daily search obtaining a result every day with all the information you need and then save this result in a Summary Index, consuming a really negligible amount of disk space and having the info that you they serve.
You can do this using the collect command (see https://docs.splunk.com/Documentation/Splunk/7.3.0/SearchReference/Collect).
Then you can have these information searching on this Summary index.

Bye.
Giuseppe

Re: Creating a lookup file to provide data to a dashboard search with time stamps

arrcee — Fri, 12 Jul 2019 22:30:15 GMT

I have not. Thank you for the suggestion. I'll look into that.

Re: Creating a lookup file to provide data to a dashboard search with time stamps

woodcock — Mon, 15 Jul 2019 03:03:46 GMT

You are mistaken; you can do a time-based lookup and that is exactly what you should do:
https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Defineatime-basedlookupinSplunkWeb

Re: Creating a lookup file to provide data to a dashboard search with time stamps

arrcee — Mon, 15 Jul 2019 23:04:23 GMT

I will look into this option as well. Thank You!

Re: Creating a lookup file to provide data to a dashboard search with time stamps

woodcock — Mon, 15 Jul 2019 23:10:12 GMT

Be sure to come back here and let us know what you ended up doing and close out the question.