Include source file that ended with date (not bz2)

[indeed_2000]

Need to declare in spl Include only those file that has ended with date not .bz2 (I don’t want to use  NOT)

Here is spl:

index="myindex" source="/data/app/20211209/CUS/app.log.*" | dedup source| table source

Return:

/data/app/20211209/CUS/app.log.2021-12-09.bz2

/data/app/20211209/CUS/app.log.2021-12-09

I try below spl but doesn’t return result

source="/data/app/20211209/CUS/app.log.*.”

Any idea?

Thanks

[ldongradi_SPL]

There is no problem using NOT source=*bz2 ... or source !=*.bz2 assuming you mention source=...

Adding an extra filter after a pipe with any where/search/regex option would not give you the best performance.

If this is a real problem, then consider indexing the bz2 files in another index.

You should also find your data by narrowing down to the sourcetype, no ?

[PickleRick]

Unfortunately, the filter part of the search does not in itself provide support for regexes, so you can only match on simple wildcards.

But, since you're only interested in the source field values, it's way, way more effective to use tstats instead of simple search and statsing.

For example

| tstats values(source) as source where index=myindex"
| mvexpand source
| regex source="/data/app/.*\d{4}-\d{2}-\d{2}$"

 In general case - @richgalloway 's soultion is quite ok. It's just that in this particular case |tstats is way faster.

[richgalloway]

A regular expression should do the job.  Use a wildcard in the base search to select events, then narrow the selection using regex.

index="myindex" source="/data/app/20211209/CUS/app.log.*" 
| regex source="app\.log\.\d{4}-\d\d-\d\d$"
| dedup source
| table source