Getting Data In

Help filtering data to nullQueue

johnward4
Communicator

I'm trying to filter out unwanted data but it's not working using my current stanzas in props & transforms. However, I was able to filter using the regex and reset the sourcetype so that should rule out an issue with the regex I'm attempting to use..

sample_log for applicationone :

2019-12-03 00:59:57,812  stdout INFO [ajp-/0.0.0.0:8009-16]: Hibernate: select sample.SAMPLE_ID as SAMPLE_ID1_5_, SAMPLE0_.sample_DESCRIPTION as sample_DESCRIPTI2_5_ from sample_SAMPLE functional0_ 

props.conf

[applicationone:log]
TRANSFORMS-sendtonull = removeDBqueries

transforms.conf

[removeDBqueries]
REGEX = select\s+.*)
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
1 Solution

woodcock
Esteemed Legend

Fix this:

 REGEX = select\s+.*\)

View solution in original post

0 Karma

woodcock
Esteemed Legend

Fix this:

 REGEX = select\s+.*\)
0 Karma

johnward4
Communicator

there was an issue with my REGEX. This did the trick:

REGEX = (SELECT|Select|select)\s+
DEST_KEY = queue
FORMAT = nullQueue

gcusello
Esteemed Legend

Hi @johnward4,
two questions:

  • where are you executing this filter? you can do it only on Indexers or (when present) on Heavy Forwarders;
  • what's "applicationone:log" that you use in the stanza's title in props.conf? usually it's used sourcetype (better) or host or source.

Ciao.
Giuseppe

0 Karma

johnward4
Communicator

Right now, I'm building the add-on in my single instance test environment.

"applicationone:log" is the name I picked for the data sourcetype.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Can you please remove bracket from REGEX and check ? Like REGEX = select\s+.*

0 Karma

johnward4
Communicator

@harsmarvania57 I tried that and it still isn't working. Could it be a problem with the sourcetype I using, does it need to be applied to _raw log data?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Can you please confirm on which instance you have applied above configuration ? It must be on Indexer or Heavy Forwarder, whichever comes first from Universal Forwarder.

0 Karma

johnward4
Communicator

I'm trying this on a single test instance. After I make a change to my configs, I delete the data from the index and restart the instance. I then upload the data again to apply my updated configs against it.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Sourcetype should work, and that REGEX will apply to _raw. Have you restarted splunk after changing config? Additionally only new data will go to nullQueue based on REGEX match, old data will stay.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...