Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

How to send some syslog messages to nullQueue - naive config not working

danbah
New Member
‎04-05-2020 01:42 PM

Running Enterprise 8.0.2.1. Data is coming in from a universal forwarder with index=syslog sourcetype=syslog and I'm trying to filter out unwanted messages. Here's a sample of the data:

2020-04-05T20:06:41.435487+00:00 HOST123 2020-04-05 20:06:41,424 Level="INFO" Name="support.bfcp" Message="Received BFCP message" Dst-address="x.x.x.x" Dst-port="41890" Src-address="y.y.y.y" Src-port="28888" Call-id="00000000-1111-2222-3333-444444444444" Primitive="Hello" Transaction-id="1014"
2020-04-05T20:06:37.552312+00:00 HOST123 2020-04-05 20:06:37,551 Level="INFO" Name="support.ice" Message="ICE new-local-candidate event" Media-type="h224" Stream-id="4" Component-id="RTCP" Local-candidate-type="host" Local-candidate-address="x.x.x.x" Local-candidate-port="41659" Local-candidate-transport="udp" Call-id="None"
2020-04-05T20:09:08.286431+00:00 HOST123 2020-04-05 20:09:08,269 Level="INFO" Name="support.participant" Message="Media Stream created" Participant="Patient" Call-id="00000000-1111-2222-3333-444444444444" Conversation-id="00000000-1111-2222-3333-444444444444" Detail="Stream 1 (video)"

I want to send certain events to nullQueue based on the Name="blah" field, so I naively did the following on the indexer:

/opt/splunk/etc/system/local/props.conf:

[syslog]
TRANSFORMS-mysystem = mysystem-nullqueue

/opt/splunk/etc/system/local/transforms.conf:

[mysystem-nullqueue]
DEST_KEY = queue
REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
FORMAT = nullQueue

Output of splunk cmd btool XXX list --debug for XXX=transforms/props:

/opt/splunk/etc/system/local/transforms.conf                           [mysystem-nullqueue]
/opt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf                         DEPTH_LIMIT = 1000
/opt/splunk/etc/system/local/transforms.conf                           DEST_KEY = queue
/opt/splunk/etc/system/local/transforms.conf                           FORMAT = nullQueue
/opt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/opt/splunk/etc/system/local/transforms.conf                           REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
/opt/splunk/etc/system/default/transforms.conf                         SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                         WRITE_META = False

/opt/splunk/etc/apps/search/local/props.conf                      [syslog]
/opt/splunk/etc/system/default/props.conf                         ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/props.conf                      EXTRACT-mysystem-syslog-apache = apache2.\d+.: (?<srcip>\S+).*?\"(?<method>\S+) (?<url>[^ ?]+)\?*(?<query>\S*) \S+\" \S+ (?<respcode>\d+) (?<respbytes>\S+) (?<resptime>\d+)
/opt/splunk/etc/apps/search/local/props.conf                      FIELDALIAS-syslog_dst_address = Dst_address ASNEW dest Dst_port ASNEW dest_port Src_address ASNEW src Src_port ASNEW src_port
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE =
/opt/splunk/etc/system/default/props.conf                         LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 32
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf                         REPORT-syslog = syslog-extractions
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf                         TIME_FORMAT = %b %d %H:%M:%S
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = syslog-host
/opt/splunk/etc/system/local/props.conf                           TRANSFORMS-mysystem = mysystem-nullqueue
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                         category = Operating System
/opt/splunk/etc/system/default/props.conf                         description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 3
/opt/splunk/etc/system/default/props.conf                         priority =
/opt/splunk/etc/system/default/props.conf                         pulldown_type = true
/opt/splunk/etc/system/default/props.conf                         sourcetype =

After a config refresh or a restart of Splunk, the syslog index is still adding new entries containing Name="support.rest" or Name="support.ice". How do I further debug nullQueue not working?

0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 02:26 PM

your btool output:
/opt/splunk/etc/apps/search/local/props.conf [syslog]
This is not /opt/splunk/etc/system/local/props.conf:
but
/opt/splunk/etc/system/local/props.conf TRANSFORMS-mysystem = mysystem-nullqueue
something is wrong.

0 Karma
Reply

danbah
New Member
‎04-05-2020 02:30 PM

Yeah I'm assuming that etc/apps/search/local is there because a field lookup was added for sourcetype=syslog in the Splunk UI, referenced by EXTRACT-mysystem-syslog-apache and FIELDALIAS-syslog_dst_address lines.

0 Karma
Reply

danbah
New Member
‎04-05-2020 03:23 PM

Doesn't that precedence only apply to configuration items that appear in multiple locations, and thus need to be overridden in some defined order?

That is, even if app config is higher priority here, there's no TRANSFORMS or TRANSFORMS-* items to process app-wise, thus we get the debug output showing that the active config items are TRANSFORMS = syslog-host and TRANSFORMS-mysystem = mysystem-nullqueue.

0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 04:11 PM

Why not do you write props.conf with nullqueue under etc/apps?

Doesn't that precedence only apply to configuration items that appear in multiple locations, and thus need to be overridden in some defined order?
I don't know how it works. I hope someone answers.

0 Karma
Reply

danbah
New Member
‎04-05-2020 04:19 PM

Restarting after moving/duplicating the settings into etc/apps/search/local/ files still doesn't filter out any of the syslog data.

0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 04:29 PM

how 's btool output?
new syslog message doesn't filter out?

0 Karma
Reply

danbah
New Member
‎04-05-2020 05:20 PM

The btool output is the same as before, but the "TRANSFORMS-mysystem = mysystem-nullqueue" line is showing as coming from the search files instead of the etc/system/local files, same with the transforms.conf header/dest_key/format/regex lines. All the unwanted syslog messages are still making it through to the index.

0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 06:08 PM

btool output:

 /opt/splunk/etc/apps/search/local/props.conf  [syslog]
 /opt/splunk/etc/apps/search/local/props.conf  TRANSFORMS-mysystem = mysystem-nullqueue

These have to be this.

0 Karma
Reply

danbah
New Member
‎04-05-2020 07:17 PM

Current config after restart. The unwanted new messages are still being indexed. Is Splunk just not parsing these events for some reason? Is there any way to check and see if this syslog traffic is actually going through the right internal Splunk queues?

splunk cmd btool transforms list --debug :

/opt/splunk/etc/apps/search/local/transforms.conf                      [mysystem-nullqueue]
/opt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf                         DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/transforms.conf                      DEST_KEY = queue
/opt/splunk/etc/apps/search/local/transforms.conf                      FORMAT = nullQueue
/opt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/opt/splunk/etc/apps/search/local/transforms.conf                      REGEX = Name="support\.(ice|rest|dns|h323|bfcp|sip)"
/opt/splunk/etc/system/default/transforms.conf                         SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                         WRITE_META = False

splunk cmd btool props list --debug :

/opt/splunk/etc/apps/search/local/props.conf                      [syslog]
/opt/splunk/etc/system/default/props.conf                         ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/props.conf                      EXTRACT-mysystem-syslog-apache = apache2.\d+.: (?<srcip>\S+).*?\"(?<method>\S+) (?<url>[^ ?]+)\?*(?<query>\S*) \S+\" \S+ (?<respcode>\d+) (?<respbytes>\S+) (?<resptime>\d+)
/opt/splunk/etc/apps/search/local/props.conf                      FIELDALIAS-syslog_dst_address = Dst_address ASNEW dest Dst_port ASNEW dest_port Src_address ASNEW src Src_port ASNEW src_port
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE =
/opt/splunk/etc/system/default/props.conf                         LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 32
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf                         REPORT-syslog = syslog-extractions
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf                         TIME_FORMAT = %b %d %H:%M:%S
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = syslog-host
/opt/splunk/etc/apps/search/local/props.conf                      TRANSFORMS-mysystem = mysystem-nullqueue
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                         category = Operating System
/opt/splunk/etc/system/default/props.conf                         description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 3
/opt/splunk/etc/system/default/props.conf                         priority =
/opt/splunk/etc/system/default/props.conf                         pulldown_type = true
/opt/splunk/etc/system/default/props.conf                         sourcetype =
0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 07:36 PM

before: REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
now: REGEX = Name="support\.(ice|rest|dns|h323|bfcp|sip)"

simply:

REGEX = support\.(ice|rest|dns|h323|bfcp|sip)

How about this?
When REGEX matches the part of event , The event is null.

0 Karma
Reply

danbah
New Member
‎04-05-2020 07:44 PM

I've tried that, and replacing the regex string with just "REGEX = bfcp" or "REGEX = .bfcp." to try to eliminate the single set of events, but no matter what they keep being indexed.

My assumption is that the forwarder is actually a heavy forwarder so Splunk will not reparse the incoming data. Unfortunately the indexer is not under my control so I'm not sure how to find out what kind of data it's giving me.

0 Karma
Reply

to4kawa
Ultra Champion
‎04-05-2020 08:19 PM

indexer and HF need same props.conf(null queue.)

https://answers.splunk.com/answers/458237/setting-up-propsconf-at-the-heavy-forwarders.html

0 Karma
Reply

danbah
New Member
‎04-05-2020 09:05 PM

So if i don't control the forwarder and can't make changes on it, there's no way to have the indexer filter out these events?

0 Karma
Reply

to4kawa
Ultra Champion
‎04-06-2020 12:40 AM

sorry, I don't know.

0 Karma
Reply
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...