topic Re: Include source file that ended with date (not bz2) in Splunk Search

Include source file that ended with date (not bz2)

indeed_2000 — Fri, 10 Dec 2021 10:05:34 GMT

Need to declare in spl Include only those file that has ended with date not .bz2 (I don’t want to use NOT)

Here is spl:

index="myindex" source="/data/app/20211209/CUS/app.log.*" | dedup source| table source

Return:

/data/app/20211209/CUS/app.log.2021-12-09.bz2

/data/app/20211209/CUS/app.log.2021-12-09

I try below spl but doesn’t return result

source="/data/app/20211209/CUS/app.log.*.”

Any idea?

Thanks

richgalloway — Fri, 10 Dec 2021 13:09:20 GMT

A regular expression should do the job. Use a wildcard in the base search to select events, then narrow the selection using regex.

index="myindex" source="/data/app/20211209/CUS/app.log.*" | regex source="app\.log\.\d{4}-\d\d-\d\d$" | dedup source | table source

PickleRick — Fri, 10 Dec 2021 13:19:14 GMT

Unfortunately, the filter part of the search does not in itself provide support for regexes, so you can only match on simple wildcards.

But, since you're only interested in the source field values, it's way, way more effective to use tstats instead of simple search and statsing.

For example

| tstats values(source) as source where index=myindex"
| mvexpand source
| regex source="/data/app/.*\d{4}-\d{2}-\d{2}$"

In general case - @richgalloway 's soultion is quite ok. It's just that in this particular case |tstats is way faster.

ldongradi_SPL — Fri, 10 Dec 2021 14:56:33 GMT

There is no problem using NOT source=*bz2 ... or source !=*.bz2 assuming you mention source=...

Adding an extra filter after a pipe with any where/search/regex option would not give you the best performance.

If this is a real problem, then consider indexing the bz2 files in another index.

You should also find your data by narrowing down to the sourcetype, no ?