Getting Data In

filter data by source field wildcard

Shakira1
Explorer

I've use case that I need to filter data by source field, that always changes.

in the transforms.conf I use:

[foo]
REGEX = MY REGEX
DEST_KEY = queue
FORMAT = nullQueue 

and in the props.conf I use:

[source::process_events]
TRANSFORMS-01= foo

The source always contains process_events and there is more data like date and info that changed. 

any way its possible to filter data by source wildcard?

thanks!

0 Karma

Shakira1
Explorer

I still need your help pls.

its not working..

0 Karma

Shakira1
Explorer
0 Karma

Shakira1
Explorer

which mean I can use 

[source::.*process_events.*] and it's should be working? 

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Shakira1 how is your complete source looks like, if you have /\ in source that might not work.

0 Karma

Shakira1
Explorer
0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Shakira1 

Can you try this props.conf

[source::s3:\/\/*process_events...]

--

An upvote would be appreciated and Accept Solution if it helps!!

0 Karma

Shakira1
Explorer

I'm still getting results...

any ideas why?

Tags (1)
0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Shakira1 REGEX in your transforms conf might be not correct. Can you share sample event and your transforms, props config?

0 Karma

Shakira1
Explorer

I can't share sample because it's with PII. 

but what I want to exclude is some path in the raw data 

[foo]

so I just put the regex like that: XXX\/XXX\/XXX\/XXXX 

and in the props.conf I just add: TRANSFORMS = foo

 

 

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Shakira1  can you try this, REGEX matches 4 segment dir structure in your _raw event. The following conf shall be deployed to HF/indexers.

 

# props.conf
[source::s3:\/\/*process_events...]
TRANSFORMS-nullq = sendtonull

#transforms.conf
[sendtonull]
REGEX = [\w-]+\/[\w-]+\/[\w-]+\/[\w-]+
FORMAT = nullQueue
DEST_KEY = queue

 

 ---

An upvote would be appreciated and Accept solution if it helps!

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@Shakira1 

If i understand correctly you have source= <values> having multiple combinations which you want to use in props.conf the source might always contains process_events?

Example. process_events_26062021,  process_events_27062021, log_process_events_26062021

It is possible to match them using Regex style- 

#your props.conf can be - 

[source::<regex>] 

TRANSFORMS-01= foo

Docs says , - Refer - https://docs.splunk.com/Documentation/Splunk/latest/Admin/PropsConf

 

When setting a [<spec>] stanza, you can use the following regex-type syntax:
... recurses through directories until the match is met
    or equivalently, matches any number of characters.
*   matches anything but the path separator 0 or more times.
    The path separator is '/' on unix, or '\' on Windows.
    Intended to match a partial or complete directory or filename.
|   is equivalent to 'or'
( ) are used to limit scope of |.
\\ = matches a literal backslash '\'.

Example: [source::....(?<!tar.)(gz|bz2)]

  

---

An upvote would be appreciated and Accept Solution if it helps!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...