How to write a search to only show the latest contents of a lookup file on a dashboard?

[kuga_mbsd]

Hi there,

My external program is retrieving the data and creating lookup table every night. The files are stored like below.
$SPLUNK_HOME/etc/apps/MyApp/lookups/FILE_YYYYmmdd.

The thing is I have to check the lookup table manually and give the list to my colleague every time since they don't know SPL at all.
Is there any good SPL to display only the latest contents of lookup flle on dashboard?

Thanks in advance.

[Lucas_K · Accepted solution]

SPL can do it easy.

2 ways. You can either have it copy the data OR have the macro fill in the name based on an eval.

1. You will have both the old with old filename and also a newer one that can always be accessed with the same name/definition.

Just do an

inputlookup `latest_lookup_file` | outputlookup latestlookup

"latest_lookup_file" is macro that will figure out the file format that you specified automatically.
latestlookup should be a definition pointing to another csv file that users will always access.

ie. macros.conf

[latest_lookup_file]
definition = lower(strftime(relative_time(time(), "@d"), "FILE_%Y%m%d.csv"))
iseval = 1

Now schedule that inputlookup line as a savedsearch that is run some time after the your file is scheduled to be uploaded.

or 2. Use the macro from above and just call it directly.

inputlookup `latest_lookup_file`

That will automatically find the latest file. There won't be a copy but all your old ones will be there. The downside to this is that if your script doesn't run anything that uses this macro will fail the next day. Method 1 is safest.

Enjoy!

[joxley]

That's really clever! I love that you're using strftime to generate the filename 🙂

[kuga_mbsd]

Thank you for your comment, Lucas.

Seems like it is doable with using macros.conf!
I've never configured this file but worth to try.

Thanks a bunch!!

[DMohn]

Have you considered rotating the "old" lookup files instead? By this you wouldn't have to check a new lookup file every day.

1. Copy lookup.csv file to lookup_current_date.csv
2. Overwrite lookup.csv

You don't have to change anything in the SPL or in your dahsboards this way.

[kuga_mbsd]

Thank you very much for your reply, DMohn.

I think your idea sounds good, however, I'm afraid that it doesnt work for my case because I need to keep the date of file creation at the file name for the records.

Please advise.

[DMohn]

Is the file name "for the records" needed within the lookup as well?

Otherwise you could go for creating two files with your external data source: the lookup.csv which will be used by Splunk dashboard, and a FILE_YYYYmmdd, as you do currently. Furthermore, you could use a small script, which lists the contents of your lookup directory, splunk the results, and display the latest file name on your dashboard for reference.

[kuga_mbsd]

well... yes, I was thinking exact your suggestions, to create two files or make script.
I was wondering if I could make it by SPL, but it seems undoable...

Anyway, thanks a lot, DMohn!