Splunk Search

Why isn't this regex working on /var/log?

rafamss
Contributor

Hi,

I'm using a Single Instance of Splunk 6.6.2 and I've tried filtering some events of my log using the code below, but the filter doesn't work. I put this argument "[\dbus]" into regex because I don't want this to be indexed. What's wrong with this?

inputs.conf:

[source::/var/log/messages]
disabled = 0
index = main
sourcetype = my_sourcetype

props.conf:

[my_sourcetype]
TRANSFORMS-null = setnull

transforms.conf:

[setnull]
REGEX = \[dbus\]
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
1 Solution

rafamss
Contributor

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

View solution in original post

rafamss
Contributor

Hi Everyone,
All this options above did help me to do the configuration that worked in my environment. Below, follow what I did.

inputs.conf:

[monitor:///var/log]
disabled = false
sourcetype = my_sourcetype
index = main

props.conf:

[my_sourcetype]
TRANSFORMS = null_queue_filter

transforms.conf:

[null_queue_filter]
REGEX = .dbus.
DEST_KEY = queue
FORMAT = nullQueue

Thank all!

woodcock
Esteemed Legend

Be sure to UpVote every helpful answer and comment.

rafamss
Contributor

Done @woodcock!

saurabh_tek11
Communicator
  1. Make sure you have corrected your inputs.conf file to monitor

  2. At transforms.conf - you might need to correct the regex to

    REGEX = dbus(\-|\[)*

(here you are matching with either dbus-* OR dbus[*)

Carefully have a look on your raw data - there is no exact word "dbus" as your regex suggests, there is "dbus*". To be precise - its either "dbus-" OR "dbus["

I would suggest you to make this change and then restart your single instance splunk to verify the results.

Correct me if i am wrong. I would wait for your feedback. Thanks,
Saurabh

saurabh_tek11
Communicator

@rafamss - Does this help?

rafamss
Contributor

Yes, it does. Thank you!

0 Karma

woodcock
Esteemed Legend

I suspect that you are copying too literally from the example docs here:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

The example there shows this:

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

But that is because it is trying to match the EXACT string [sshd]. You are probably trying to match the exact string dbus so you should use this:

REGEX = dbus

rafamss
Contributor

Is exactly that @woodcock (I followed this example). But even puting dbus or [dbus] into REGEX option, this still not work.

0 Karma

woodcock
Esteemed Legend

Make sure that your sourcetype is right in the stanza header. Make sure that you deploy the file to the indexers (or Heavy Forwarders). Make sure that you restart all Splunk instances there. Make sure that you verify/test by looking at events that were forwarded in after the restarts (previously indexed events do NOT get re-examines and deleted).

0 Karma

jplumsdaine22
Influencer

What behaviour are you expecting? The transform you posted will dump every event that contains the string [dbus]. If the splunk search sourcetype=my_sourcetype TERM("\[dbus\]") returns no events then its working.

Where have you put the props and transforms file? They must be on the indexer. Also you need to restart splunk after updating the transform.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you share some sample raw data that you want to drop? (mask any sensitive information)

rafamss
Contributor

Sure @somesoni2.

Nov 28 18:02:53 localhost dbus-daemon: dbus[409]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[809]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus-daemon: dbus[981]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'Nov 28 18:02:53 localhost dbus-daemon: dbus[604]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[605]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 28 18:02:53 localhost dbus[600]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So you want to drop any event which has word dbus? Or is it dbus[? If that's the case, your REGEX in your transforms.conf should be this REGEX = dbus OR REGEX = dbus\[ for second case.

The current value of REGEX = \[dbus\] actually looks for literal string [dbus] in the events, which I don't see in your sample data, hence it didn't work.

rafamss
Contributor

Is exactly what I want to do. Drop all events with dbus and store the events that not have this parameter. I'll test your sample and go back here.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Just to verify - each key word is on a line by itself, true?

  [setnull] 
  REGEX = \[dbus\] 
  DEST_KEY = queue 
  FORMAT = nullQueue

rafamss
Contributor

Each key is on a your line. The code style of answers that put all into a single line @DalJeanis.

0 Karma

nileena
Path Finder

The inputs stanza should be

[monitor:///var/log/messages]

Are there any internal errors you see when you search "index=_internal"?

rafamss
Contributor

As a fact @nileena. In my environment I put the stanza like as below. In the internal index don't have any error that contains references to this.

[monitor:///var/log/messages]
disabled = false
index = main
sourcetype = my_sourcetype

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...