Why isn't this regex working on /var/log?

[rafamss]

Hi,

I'm using a Single Instance of Splunk 6.6.2 and I've tried filtering some events of my log using the code below, but the filter doesn't work. I put this argument "[\dbus]" into regex because I don't want this to be indexed. What's wrong with this?

inputs.conf:

[source::/var/log/messages]
disabled = 0
index = main
sourcetype = my_sourcetype

props.conf:

[my_sourcetype]
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = \[dbus\]
DEST_KEY = queue
FORMAT = nullQueue

[rafamss · Accepted solution]

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

[woodcock]

Be sure to UpVote every helpful answer and comment.

[rafamss]

Done @woodcock!

[saurabh_tek11]

1. Make sure you have corrected your inputs.conf file to monitor

2. At transforms.conf - you might need to correct the regex to

   REGEX = dbus(\-|\[)*

(here you are matching with either dbus-* OR dbus[*)

Carefully have a look on your raw data - there is no exact word "dbus" as your regex suggests, there is "dbus*". To be precise - its either "dbus-" OR "dbus["

I would suggest you to make this change and then restart your single instance splunk to verify the results.

Correct me if i am wrong. I would wait for your feedback. Thanks,
Saurabh

[saurabh_tek11]

@rafamss - Does this help?

[rafamss]

Yes, it does. Thank you!

[woodcock]

I suspect that you are copying too literally from the example docs here:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

The example there shows this:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

But that is because it is trying to match the EXACT string [sshd]. You are probably trying to match the exact string dbus so you should use this:

REGEX = dbus

[rafamss]

Is exactly that @woodcock (I followed this example). But even puting dbus or [dbus] into REGEX option, this still not work.

[woodcock]

Make sure that your sourcetype is right in the stanza header. Make sure that you deploy the file to the indexers (or Heavy Forwarders). Make sure that you restart all Splunk instances there. Make sure that you verify/test by looking at events that were forwarded in after the restarts (previously indexed events do NOT get re-examines and deleted).

[jplumsdaine22]

What behaviour are you expecting? The transform you posted will dump every event that contains the string [dbus]. If the splunk search sourcetype=my_sourcetype TERM("\[dbus\]") returns no events then its working.

Where have you put the props and transforms file? They must be on the indexer. Also you need to restart splunk after updating the transform.

[somesoni2]

Can you share some sample raw data that you want to drop? (mask any sensitive information)

[rafamss]

Sure @somesoni2.

Nov 28 18:02:53 localhost dbus-daemon: dbus[409]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[809]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[981]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'Nov 28 18:02:53 localhost dbus-daemon: dbus[604]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[605]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[600]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'

[somesoni2]

So you want to drop any event which has word dbus? Or is it dbus[? If that's the case, your REGEX in your transforms.conf should be this REGEX = dbus OR REGEX = dbus\[ for second case.

The current value of REGEX = \[dbus\] actually looks for literal string [dbus] in the events, which I don't see in your sample data, hence it didn't work.

[rafamss]

Is exactly what I want to do. Drop all events with dbus and store the events that not have this parameter. I'll test your sample and go back here.

[DalJeanis]

Just to verify - each key word is on a line by itself, true?

  [setnull] 
  REGEX = \[dbus\] 
  DEST_KEY = queue 
  FORMAT = nullQueue

[rafamss]

Each key is on a your line. The code style of answers that put all into a single line @DalJeanis.

[nileena]

The inputs stanza should be

[monitor:///var/log/messages]

Are there any internal errors you see when you search "index=_internal"?

[rafamss]

As a fact @nileena. In my environment I put the stanza like as below. In the internal index don't have any error that contains references to this.

[monitor:///var/log/messages]
disabled = false
index = main
sourcetype = my_sourcetype