topic Re: Why isn't this regex working on /var/log? in Splunk Search

Why isn't this regex working on /var/log?

rafamss — Tue, 28 Nov 2017 17:44:13 GMT

Hi,

I'm using a Single Instance of Splunk 6.6.2 and I've tried filtering some events of my log using the code below, but the filter doesn't work. I put this argument "[\dbus]" into regex because I don't want this to be indexed. What's wrong with this?

inputs.conf:

[source::/var/log/messages]
disabled = 0
index = main
sourcetype = my_sourcetype

props.conf:

[my_sourcetype]
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = \[dbus\]
DEST_KEY = queue
FORMAT = nullQueue

Re: Why isn't this regex working on /var/log?

nileena — Tue, 28 Nov 2017 17:53:54 GMT

The inputs stanza should be

[monitor:///var/log/messages]

Are there any internal errors you see when you search "index=_internal"?

Re: Why isn't this regex working on /var/log?

DalJeanis — Tue, 28 Nov 2017 18:42:39 GMT

Just to verify - each key word is on a line by itself, true?

  [setnull] 
  REGEX = \[dbus\] 
  DEST_KEY = queue 
  FORMAT = nullQueue

Re: Why isn't this regex working on /var/log?

rafamss — Tue, 28 Nov 2017 18:43:49 GMT

As a fact @nileena. In my environment I put the stanza like as below. In the internal index don't have any error that contains references to this.

[monitor:///var/log/messages] disabled = false index = main sourcetype = my_sourcetype

Re: Why isn't this regex working on /var/log?

rafamss — Tue, 28 Nov 2017 18:59:20 GMT

Each key is on a your line. The code style of answers that put all into a single line @DalJeanis.

Re: Why isn't this regex working on /var/log?

somesoni2 — Tue, 28 Nov 2017 19:09:02 GMT

Can you share some sample raw data that you want to drop? (mask any sensitive information)

Re: Why isn't this regex working on /var/log?

rafamss — Tue, 29 Sep 2020 16:58:17 GMT

Sure @somesoni2.

Nov 28 18:02:53 localhost dbus-daemon: dbus[409]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[809]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[981]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'Nov 28 18:02:53 localhost dbus-daemon: dbus[604]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[605]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[600]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'

Re: Why isn't this regex working on /var/log?

somesoni2 — Tue, 28 Nov 2017 20:10:48 GMT

So you want to drop any event which has word dbus? Or is it dbus[? If that's the case, your REGEX in your transforms.conf should be this REGEX = dbus OR REGEX = dbus\[ for second case.

The current value of REGEX = \[dbus\] actually looks for literal string [dbus] in the events, which I don't see in your sample data, hence it didn't work.

Re: Why isn't this regex working on /var/log?

woodcock — Wed, 29 Nov 2017 00:47:10 GMT

I suspect that you are copying too literally from the example docs here:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

The example there shows this:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

But that is because it is trying to match the EXACT string [sshd]. You are probably trying to match the exact string dbus so you should use this:

REGEX = dbus

Re: Why isn't this regex working on /var/log?

rafamss — Wed, 29 Nov 2017 12:17:53 GMT

Is exactly that @woodcock (I followed this example). But even puting dbus or [dbus] into REGEX option, this still not work.

Re: Why isn't this regex working on /var/log?

rafamss — Wed, 29 Nov 2017 12:39:13 GMT

Is exactly what I want to do. Drop all events with dbus and store the events that not have this parameter. I'll test your sample and go back here.

Re: Why isn't this regex working on /var/log?

jplumsdaine22 — Wed, 29 Nov 2017 12:55:22 GMT

What behaviour are you expecting? The transform you posted will dump every event that contains the string [dbus]. If the splunk search sourcetype=my_sourcetype TERM("\[dbus\]") returns no events then its working.

Where have you put the props and transforms file? They must be on the indexer. Also you need to restart splunk after updating the transform.

Re: Why isn't this regex working on /var/log?

saurabh_tek11 — Tue, 29 Sep 2020 16:58:37 GMT

Make sure you have corrected your inputs.conf file to monitor
At transforms.conf - you might need to correct the regex to

REGEX = dbus(\-|\[)*

(here you are matching with either dbus-* OR dbus[*)

Carefully have a look on your raw data - there is no exact word "dbus" as your regex suggests, there is "dbus*". To be precise - its either "dbus-" OR "dbus["

I would suggest you to make this change and then restart your single instance splunk to verify the results.

Correct me if i am wrong. I would wait for your feedback. Thanks,
Saurabh

Re: Why isn't this regex working on /var/log?

woodcock — Wed, 29 Nov 2017 13:12:44 GMT

Make sure that your sourcetype is right in the stanza header. Make sure that you deploy the file to the indexers (or Heavy Forwarders). Make sure that you restart all Splunk instances there. Make sure that you verify/test by looking at events that were forwarded in after the restarts (previously indexed events do NOT get re-examines and deleted).

Re: Why isn't this regex working on /var/log?

saurabh_tek11 — Thu, 30 Nov 2017 19:07:24 GMT

@rafamss - Does this help?

Re: Why isn't this regex working on /var/log?

rafamss — Mon, 04 Dec 2017 15:31:55 GMT

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

Re: Why isn't this regex working on /var/log?

woodcock — Mon, 04 Dec 2017 15:37:32 GMT

Be sure to UpVote every helpful answer and comment.

Re: Why isn't this regex working on /var/log?

rafamss — Mon, 04 Dec 2017 18:54:27 GMT

Done @woodcock!

Re: Why isn't this regex working on /var/log?

rafamss — Mon, 28 Oct 2019 21:36:19 GMT

Yes, it does. Thank you!