Splunk Search

Why isn't this regex working on /var/log?

rafamss
Contributor

Hi,

I'm using a Single Instance of Splunk 6.6.2 and I've tried filtering some events of my log using the code below, but the filter doesn't work. I put this argument "[\dbus]" into regex because I don't want this to be indexed. What's wrong with this?

inputs.conf:

[source::/var/log/messages]
disabled = 0
index = main
sourcetype = my_sourcetype

props.conf:

[my_sourcetype]
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = \[dbus\]
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
1 Solution

rafamss
Contributor

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

View solution in original post

rafamss
Contributor

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

View solution in original post

woodcock
Esteemed Legend

Be sure to UpVote every helpful answer and comment.

rafamss
Contributor

Done @woodcock!

saurabh_tek11
Communicator
  1. Make sure you have corrected your inputs.conf file to monitor

  2. At transforms.conf - you might need to correct the regex to

    REGEX = dbus(\-|\[)*

(here you are matching with either dbus-* OR dbus[*)

Carefully have a look on your raw data - there is no exact word "dbus" as your regex suggests, there is "dbus*". To be precise - its either "dbus-" OR "dbus["

I would suggest you to make this change and then restart your single instance splunk to verify the results.

Correct me if i am wrong. I would wait for your feedback. Thanks,
Saurabh

saurabh_tek11
Communicator

@rafamss - Does this help?

rafamss
Contributor

Yes, it does. Thank you!

0 Karma

woodcock
Esteemed Legend

I suspect that you are copying too literally from the example docs here:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

The example there shows this:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

But that is because it is trying to match the EXACT string [sshd]. You are probably trying to match the exact string dbus so you should use this:

REGEX = dbus

rafamss
Contributor

Is exactly that @woodcock (I followed this example). But even puting dbus or [dbus] into REGEX option, this still not work.

0 Karma

woodcock
Esteemed Legend

Make sure that your sourcetype is right in the stanza header. Make sure that you deploy the file to the indexers (or Heavy Forwarders). Make sure that you restart all Splunk instances there. Make sure that you verify/test by looking at events that were forwarded in after the restarts (previously indexed events do NOT get re-examines and deleted).

0 Karma

jplumsdaine22
Influencer

What behaviour are you expecting? The transform you posted will dump every event that contains the string [dbus]. If the splunk search sourcetype=my_sourcetype TERM("\[dbus\]") returns no events then its working.

Where have you put the props and transforms file? They must be on the indexer. Also you need to restart splunk after updating the transform.

0 Karma

somesoni2
Revered Legend

Can you share some sample raw data that you want to drop? (mask any sensitive information)

rafamss
Contributor

Sure @somesoni2.

Nov 28 18:02:53 localhost dbus-daemon: dbus[409]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[809]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[981]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'Nov 28 18:02:53 localhost dbus-daemon: dbus[604]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[605]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[600]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'

0 Karma

somesoni2
Revered Legend

So you want to drop any event which has word dbus? Or is it dbus[? If that's the case, your REGEX in your transforms.conf should be this REGEX = dbus OR REGEX = dbus\[ for second case.

The current value of REGEX = \[dbus\] actually looks for literal string [dbus] in the events, which I don't see in your sample data, hence it didn't work.

rafamss
Contributor

Is exactly what I want to do. Drop all events with dbus and store the events that not have this parameter. I'll test your sample and go back here.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Just to verify - each key word is on a line by itself, true?

  [setnull] 
  REGEX = \[dbus\] 
  DEST_KEY = queue 
  FORMAT = nullQueue

rafamss
Contributor

Each key is on a your line. The code style of answers that put all into a single line @DalJeanis.

0 Karma

nileena
Path Finder

The inputs stanza should be

[monitor:///var/log/messages]

Are there any internal errors you see when you search "index=_internal"?

rafamss
Contributor

As a fact @nileena. In my environment I put the stanza like as below. In the internal index don't have any error that contains references to this.

[monitor:///var/log/messages]
disabled = false
index = main
sourcetype = my_sourcetype

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!