topic Re: filter data by source field wildcard in Getting Data In

filter data by source field wildcard

Shakira1 — Tue, 29 Jun 2021 09:40:46 GMT

I've use case that I need to filter data by source field, that always changes.

in the transforms.conf I use:

[foo]
REGEX = MY REGEX
DEST_KEY = queue
FORMAT = nullQueue

and in the props.conf I use:

[source::process_events]
TRANSFORMS-01= foo

The source always contains process_events and there is more data like date and info that changed.

any way its possible to filter data by source wildcard?

thanks!

Re: filter data by source field wildcard

venkatasri — Tue, 29 Jun 2021 10:08:16 GMT

@Shakira1

If i understand correctly you have source= <values> having multiple combinations which you want to use in props.conf the source might always contains process_events?

Example. process_events_26062021, process_events_27062021, log_process_events_26062021

It is possible to match them using Regex style-

#your props.conf can be -

[source::<regex>]

TRANSFORMS-01= foo

Docs says , - Refer - https://docs.splunk.com/Documentation/Splunk/latest/Admin/PropsConf

When setting a [<spec>] stanza, you can use the following regex-type syntax: ... recurses through directories until the match is met or equivalently, matches any number of characters. * matches anything but the path separator 0 or more times. The path separator is '/' on unix, or '\' on Windows. Intended to match a partial or complete directory or filename. | is equivalent to 'or' ( ) are used to limit scope of |. \\ = matches a literal backslash '\'. Example: [source::....(?<!tar.)(gz|bz2)]

---

An upvote would be appreciated and Accept Solution if it helps!

Re: filter data by source field wildcard

Shakira1 — Tue, 29 Jun 2021 10:40:32 GMT

which mean I can use

[source::.*process_events.*] and it's should be working?

Re: filter data by source field wildcard

venkatasri — Tue, 29 Jun 2021 10:42:53 GMT

@Shakira1 how is your complete source looks like, if you have /\ in source that might not work.

Re: filter data by source field wildcard

Shakira1 — Tue, 29 Jun 2021 10:45:14 GMT

for example:

s3://XXXXprocess_events/upt_day=20210627/part-00053-a08c751e-1c05-4725-bdea-740009db640e.c000.json.gz

Re: filter data by source field wildcard

Shakira1 — Tue, 29 Jun 2021 10:51:03 GMT

for example:

s3://XXXXprocess_events/upt_day=20210627/part-00053-a08c751e-1c05-4725-bdea-740009db640e.c000.json.g...

Re: filter data by source field wildcard

Shakira1 — Tue, 29 Jun 2021 11:52:31 GMT

I still need your help pls.

its not working..

Re: filter data by source field wildcard

venkatasri — Tue, 29 Jun 2021 21:55:09 GMT

@Shakira1

Can you try this props.conf

[source::s3:\/\/*process_events...]

An upvote would be appreciated and Accept Solution if it helps!!

Re: filter data by source field wildcard

Shakira1 — Wed, 30 Jun 2021 14:21:58 GMT

I'm still getting results...

any ideas why?

Re: filter data by source field wildcard

venkatasri — Wed, 30 Jun 2021 21:19:56 GMT

@Shakira1 REGEX in your transforms conf might be not correct. Can you share sample event and your transforms, props config?

Re: filter data by source field wildcard

Shakira1 — Thu, 01 Jul 2021 12:33:23 GMT

I can't share sample because it's with PII.

but what I want to exclude is some path in the raw data

[foo]

so I just put the regex like that: XXX\/XXX\/XXX\/XXXX

and in the props.conf I just add: TRANSFORMS = foo

Re: filter data by source field wildcard

venkatasri — Fri, 02 Jul 2021 05:24:40 GMT

@Shakira1 can you try this, REGEX matches 4 segment dir structure in your _raw event. The following conf shall be deployed to HF/indexers.

# props.conf [source::s3:\/\/*process_events...] TRANSFORMS-nullq = sendtonull #transforms.conf [sendtonull] REGEX = [\w-]+\/[\w-]+\/[\w-]+\/[\w-]+ FORMAT = nullQueue DEST_KEY = queue

---

An upvote would be appreciated and Accept solution if it helps!