Getting Data In

How to eliminate events with ">Debug"

Communicator

Trying to eliminate logs that start with ">Debug". Must be missing something with my logic.
All the data has a sourcetype=diplomat:server which I believe I've done correctly in the props, and trying to utilize the nullqueue.
I did an apply cluster-bundle after creating the new apps, see it on the indexers, and even did a rolling restart of the indexers to make sure this was in effect. Still getting the logs that start with ">Debug" though,
what did I miss???

props.conf

[diplomat:server]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = ^>Debug
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Move the app from $SPLUNK_HOME/etc/master-apps/_cluster/ to $SPLUNK_HOME/etc/master-apps/ and apply the bundle again. The directory $SPLUNK_HOME/etc/master-apps/_cluster/ is a special one and should only be used to deploy config files, not apps - see https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Updatepeerconfigurations#On_the_master

cheers, MuS

View solution in original post

SplunkTrust
SplunkTrust

Move the app from $SPLUNK_HOME/etc/master-apps/_cluster/ to $SPLUNK_HOME/etc/master-apps/ and apply the bundle again. The directory $SPLUNK_HOME/etc/master-apps/_cluster/ is a special one and should only be used to deploy config files, not apps - see https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Updatepeerconfigurations#On_the_master

cheers, MuS

View solution in original post

Communicator

Appreciate all the feedback everyone. Awesome help on the Splunk Answers site.

0 Karma

Communicator

You are 100% correct. I missed this and must be an old habit.

0 Karma

SplunkTrust
SplunkTrust

Hi joesrepsolc,

Good to hear this solved the issue. Can you please accept this as the answer? Thanks

cheers, MuS

0 Karma

Communicator

UPDATE:

The regex assistance may have helped but the issue was resolve with a Splunk support call. Evidently the props.conf/transforms.conf in my app needed to be in the /opt/splunk/etc/master-apps/ folder... not one level lower in the /opt/splunk/etc/master-apps/_cluster folder. I moved the app I had built "up" one level, and instantly started to work.

WORKED:
/opt/splunk/etc/master-apps/newApp/local/props.conf

NOT WORKED:
/opt/splunk/etc/master-apps/_cluster/newApp/local/props.conf

0 Karma

Legend

Hi joesrepsolc,
where do you located you props.conf and transforms.conf?
they must be located on Indexers or Heavy Forwarders (when present), not on UniversalForwarders.

In addition use backslash when you use special chars (like >) in regexes, in other words in transforms.conf:

[setnull]
REGEX = ^\>Debug
DEST_KEY = queue
FORMAT = nullQueue

Bye.
Giuseppe

0 Karma

Communicator

Trying this now... was not aware of the special character issue with REGEX line. Thank you.

0 Karma

Motivator

@joesrepsolc - Could you please accept this answer by @gcusello so other users can find approved answer easily.

0 Karma

SplunkTrust
SplunkTrust

Hi joesrepsolc,

two things that are important in this case:

  1. The sourcetype in props.conf is case sensitive, so make sure it really matches.
  2. The regex uses > which is a special character in regex and needs escaping like this ^\>Debug

Further more use $SPLUNK_HOME/bin/splunk btool props list diplomat:server --debug on one of the indexers to validate your props.conf is being applied and not overwritten by some other app taking precedence over your app.

Hope this helps ...

cheers, MuS

0 Karma

SplunkTrust
SplunkTrust

Update:

The issue was not the regex, but the config files were at the wrong path. See this answer for the correct solution https://answers.splunk.com/answers/750050/how-to-eliminate-events-with-debug.html#answer-751623

cheers, MuS

0 Karma

Communicator

So i put the props.conf and transforms.conf in the /opt/splunk/etc/master-apps/_cluster/diplomat/local folder. and did the apply-cluster-bundle command. I see the "app" was pushed out to the indexers in the cluster (under /opt/splunk/etc/slave-apps/_cluster/diplomat/local). And even restarted the indexers manually.

Still nothing when i run this command:
/opt/splunk/bin/splunk btool props list diplomat:server --debug

And still getting the unwanted ">Debug" events ingested. What am I missing??? thanks!

Joe

0 Karma

Motivator

Is it still your question? If yes then do not make it accepted answer so it keeps in someone's eyesight.

0 Karma

Communicator

sourcetype is case-correct. Tried that btool command, and got nothing back. I see the props/transforms on the indexers now after running apply cluster-bundle command. Maybe not restarted though... doing a rolling-restart cluster-peers now. I'll report back on outcome. Thanks!

0 Karma