Splunk Search

How to write a search to only show the latest contents of a lookup file on a dashboard?

kuga_mbsd
New Member

Hi there,

My external program is retrieving the data and creating lookup table every night. The files are stored like below.
$SPLUNK_HOME/etc/apps/MyApp/lookups/FILE_YYYYmmdd.

The thing is I have to check the lookup table manually and give the list to my colleague every time since they don't know SPL at all.
Is there any good SPL to display only the latest contents of lookup flle on dashboard?

Thanks in advance.

0 Karma
1 Solution

Lucas_K
Motivator

SPL can do it easy.

2 ways. You can either have it copy the data OR have the macro fill in the name based on an eval.

  1. You will have both the old with old filename and also a newer one that can always be accessed with the same name/definition.

Just do an

inputlookup `latest_lookup_file` | outputlookup latestlookup

"latest_lookup_file" is macro that will figure out the file format that you specified automatically.
latestlookup should be a definition pointing to another csv file that users will always access.

ie. macros.conf

[latest_lookup_file]
definition = lower(strftime(relative_time(time(), "@d"), "FILE_%Y%m%d.csv"))
iseval = 1

Now schedule that inputlookup line as a savedsearch that is run some time after the your file is scheduled to be uploaded.

or 2. Use the macro from above and just call it directly.

inputlookup `latest_lookup_file`

That will automatically find the latest file. There won't be a copy but all your old ones will be there. The downside to this is that if your script doesn't run anything that uses this macro will fail the next day. Method 1 is safest.

Enjoy!

View solution in original post

Lucas_K
Motivator

SPL can do it easy.

2 ways. You can either have it copy the data OR have the macro fill in the name based on an eval.

  1. You will have both the old with old filename and also a newer one that can always be accessed with the same name/definition.

Just do an

inputlookup `latest_lookup_file` | outputlookup latestlookup

"latest_lookup_file" is macro that will figure out the file format that you specified automatically.
latestlookup should be a definition pointing to another csv file that users will always access.

ie. macros.conf

[latest_lookup_file]
definition = lower(strftime(relative_time(time(), "@d"), "FILE_%Y%m%d.csv"))
iseval = 1

Now schedule that inputlookup line as a savedsearch that is run some time after the your file is scheduled to be uploaded.

or 2. Use the macro from above and just call it directly.

inputlookup `latest_lookup_file`

That will automatically find the latest file. There won't be a copy but all your old ones will be there. The downside to this is that if your script doesn't run anything that uses this macro will fail the next day. Method 1 is safest.

Enjoy!

joxley
Path Finder

That's really clever! I love that you're using strftime to generate the filename 🙂

0 Karma

kuga_mbsd
New Member

Thank you for your comment, Lucas.

Seems like it is doable with using macros.conf!
I've never configured this file but worth to try.

Thanks a bunch!!

0 Karma

DMohn
Motivator

Have you considered rotating the "old" lookup files instead? By this you wouldn't have to check a new lookup file every day.

  1. Copy lookup.csv file to lookup_current_date.csv
  2. Overwrite lookup.csv

You don't have to change anything in the SPL or in your dahsboards this way.

0 Karma

kuga_mbsd
New Member

Thank you very much for your reply, DMohn.

I think your idea sounds good, however, I'm afraid that it doesnt work for my case because I need to keep the date of file creation at the file name for the records.

Please advise.

0 Karma

DMohn
Motivator

Is the file name "for the records" needed within the lookup as well?

Otherwise you could go for creating two files with your external data source: the lookup.csv which will be used by Splunk dashboard, and a FILE_YYYYmmdd, as you do currently. Furthermore, you could use a small script, which lists the contents of your lookup directory, splunk the results, and display the latest file name on your dashboard for reference.

0 Karma

kuga_mbsd
New Member

well... yes, I was thinking exact your suggestions, to create two files or make script.
I was wondering if I could make it by SPL, but it seems undoable...

Anyway, thanks a lot, DMohn!

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...