Splunk Search

I want collect all characters and events that end with a certain text via Regex?

Dark_Ichigo
Builder

I want to collect all data before a specified text or that ends with it, I have tried the following:

   (.+?)ABC_.*|(.+?)\>

But you have to specify how many characters you want to collect before the specified ending text, I want to just collect all of them (Note that I'm dealing with Python/Perl regexes here in Splunk)

The reason for this is to add the rest to the NullQueue while collecting these.

Update

A quick example of what I have been trying to do, but been failing at:

[ABC_setnull]
REGEX = .*
DEST_KEY = queue
FORMAT = nullQueue

[ABC_setparsing]
REGEX = (.+?)ABC_.*|(.+?)\>
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

Damien_Dallimor
Ultra Champion

Perhaps you could try something like this with a stanza in transforms.conf that will just keep the event text up to the end pattern:

props.conf

[yoursourcetype] 
TRANSFORMS-yourtransform = scooby-dooby-doo

transforms.conf

[scooby-dooby-doo] 
REGEX = (?m)^(.*)yourendpattern$ 
FORMAT = $1 
DEST_KEY = _raw

Rather than using null queues , using an anonymization based approach.

0 Karma

Damien_Dallimor
Ultra Champion

actually , 1) is what you are trying to do. 2) is altering the raw events to strip out text. 1) will route off full events you don't want to a null queue if you so desire.

In props.conf

[yoursourcetype]
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (.+?)ABC_.*|(.+?)>
DEST_KEY = queue
FORMAT = indexQueue

Can you show me an example event , perhaps your regex is wrong ?

Dark_Ichigo
Builder

Well, number (2) seems to be the one, as the main purpose of all this is to filter the data to get rid of events that I don't want indexed so that I wont reach my licensing quota, so basically if you look at my Update in the question above, you will notice that Im picking up what I want from the logs and putting it to the queue, then the rest that I dont need and have not specified goes all the way out of my way to the NullQueue.

Hope this makes sense??

0 Karma

Damien_Dallimor
Ultra Champion

I'm getting lost with your intent now.

Do you want to :

1) route raw events to different queues based on a pattern in the raw event

or

2) index only certain text from each event ( from original question : collect all data before a specified text or that ends with it )

0 Karma

Dark_Ichigo
Builder

Yep, it all makes sense, but is there a difference to saying dest_key=_raw and dest_key=Queue?

(I have modified my question)

btw, does the regex also include the end pattern along with it?

0 Karma

Damien_Dallimor
Ultra Champion

scooby-dooby-doo is an example "stanza name".
Replace with whatever you want.
In the above example the raw event(dest_key = _raw) is being transformed to only include the text before the end pattern that you specify.

0 Karma

Dark_Ichigo
Builder

Not really, I want to just filter events from a log, I thought pushing what I don't want to NullQueue might work, but its not, I have tried and tried, just now its been an hour and it seems that nothing is being indexed at all!, I'm willing to try your recommendation, so this will obviously not create a field called "scooby-dooby-doo" right?, but only index data that match the applied index (Note: the data is being forwarded to the indexer from a Forwarder).

Btw, the main reason for all this, is to filter the data of events that I dont want to be indexed....

0 Karma

Damien_Dallimor
Ultra Champion

"event-stripper" is an EXAMPLE name I made up.You could call the stanza anything you like and reference it from props.conf, adjusted example above.

Also , if I understand correctly you are trying to do more than filter and route raw events , you are trying to split up a raw event and only index part of it.

0 Karma

Dark_Ichigo
Builder

Thanks for answering, I havent heard of this before, I have read the documentation, but I cant find the "event-stripper" stanza in the transforms.conf nor the props.conf templates here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Propsconf
and here:
http://docs.splunk.com/Documentation/Splunk/5.0.2/admin/Transformsconf

So basically what your referring to here is close to the section of "Filter data and send rest to NullQueue" here: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad ???

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...