how to optymize my query

[kp_pl]

Below quite simple query to fill drop down list in my dashboard. 

index=gwcc
| eval file=lower(mvindex(split(source,"/"),-1))
| dedup file
| table source, file
| sort file

The point is it takes 30-60 seconds to generate it.   Do you have an idea how to simplify it ? Or write in more efficient way ?

[kp_pl]

metadata and with pipe at the front of .... completely new command/structure for me, but  it works, and works much faster 🙂

But one more unexpected case has appeared due to this change. I cannot filter out rotated files which are in the directory and are not necessary . It looks something like

file1.log
file1.2024-09-01.log
file1.2024-08-02.log

etc. etc.

and of course I only need the main , the most present file ( without any dates)
so I tried

| metadata type=sources where index=gwcc AND source !='*log.2024-*'
| eval source2=lower(mvindex(split(source,".2024"),-1))
| eval file=lower(mvindex(split(source,"/"),-1))
| table source, source2, file

but my "filter" does not work .

[PickleRick]

There is another way to achieve similar results. Instead of the metadata command (which is great in its own right), you can use the tstats command which might work a bit slower than metadata but can do more complicated stuff with indexed fields.

| tstats values(source) AS source WHERE index=* source !='*log.2024-*'
| mvexpand source
| <the rest of your evals>

[ITWhisperer]

The metadata command doesn't take filters other than index so filter after the data is returned

| metadata type=sources where index=gwcc
| search source !="*log.2024-*"
| eval source2=lower(mvindex(split(source,".2024"),-1))
| eval file=lower(mvindex(split(source,"/"),-1))
| table source, source2, file

[kp_pl]

great!   Works as expected 🙂

one correction:  it should be double quotes instead of single in search  🙂

| search source !="*log.2024-*"

[ITWhisperer]

Corrected

[ITWhisperer]

Try something like this

| metadata type=sources where index=gwcc
| eval file=lower(mvindex(split(source,"/"),-1))
| dedup file
| table source, file
| sort file