Running Enterprise 8.0.2.1. Data is coming in from a universal forwarder with index=syslog sourcetype=syslog and I'm trying to filter out unwanted messages. Here's a sample of the data:
2020-04-05T20:06:41.435487+00:00 HOST123 2020-04-05 20:06:41,424 Level="INFO" Name="support.bfcp" Message="Received BFCP message" Dst-address="x.x.x.x" Dst-port="41890" Src-address="y.y.y.y" Src-port="28888" Call-id="00000000-1111-2222-3333-444444444444" Primitive="Hello" Transaction-id="1014"
2020-04-05T20:06:37.552312+00:00 HOST123 2020-04-05 20:06:37,551 Level="INFO" Name="support.ice" Message="ICE new-local-candidate event" Media-type="h224" Stream-id="4" Component-id="RTCP" Local-candidate-type="host" Local-candidate-address="x.x.x.x" Local-candidate-port="41659" Local-candidate-transport="udp" Call-id="None"
2020-04-05T20:09:08.286431+00:00 HOST123 2020-04-05 20:09:08,269 Level="INFO" Name="support.participant" Message="Media Stream created" Participant="Patient" Call-id="00000000-1111-2222-3333-444444444444" Conversation-id="00000000-1111-2222-3333-444444444444" Detail="Stream 1 (video)"
I want to send certain events to nullQueue based on the Name="blah" field, so I naively did the following on the indexer:
/opt/splunk/etc/system/local/props.conf:
[syslog]
TRANSFORMS-mysystem = mysystem-nullqueue
/opt/splunk/etc/system/local/transforms.conf:
[mysystem-nullqueue]
DEST_KEY = queue
REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
FORMAT = nullQueue
Output of splunk cmd btool XXX list --debug for XXX=transforms/props:
/opt/splunk/etc/system/local/transforms.conf [mysystem-nullqueue]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/local/transforms.conf DEST_KEY = queue
/opt/splunk/etc/system/local/transforms.conf FORMAT = nullQueue
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/system/local/transforms.conf REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
/opt/splunk/etc/apps/search/local/props.conf [syslog]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/props.conf EXTRACT-mysystem-syslog-apache = apache2.\d+.: (?<srcip>\S+).*?\"(?<method>\S+) (?<url>[^ ?]+)\?*(?<query>\S*) \S+\" \S+ (?<respcode>\d+) (?<respbytes>\S+) (?<resptime>\d+)
/opt/splunk/etc/apps/search/local/props.conf FIELDALIAS-syslog_dst_address = Dst_address ASNEW dest Dst_port ASNEW dest_port Src_address ASNEW src Src_port ASNEW src_port
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 32
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf REPORT-syslog = syslog-extractions
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf TIME_FORMAT = %b %d %H:%M:%S
/opt/splunk/etc/system/default/props.conf TRANSFORMS = syslog-host
/opt/splunk/etc/system/local/props.conf TRANSFORMS-mysystem = mysystem-nullqueue
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf category = Operating System
/opt/splunk/etc/system/default/props.conf description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 3
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf pulldown_type = true
/opt/splunk/etc/system/default/props.conf sourcetype =
After a config refresh or a restart of Splunk, the syslog index is still adding new entries containing Name="support.rest" or Name="support.ice". How do I further debug nullQueue not working?
your btool output:
/opt/splunk/etc/apps/search/local/props.conf [syslog]
This is not /opt/splunk/etc/system/local/props.conf:
but
/opt/splunk/etc/system/local/props.conf TRANSFORMS-mysystem = mysystem-nullqueue
something is wrong.
Yeah I'm assuming that etc/apps/search/local is there because a field lookup was added for sourcetype=syslog in the Splunk UI, referenced by EXTRACT-mysystem-syslog-apache and FIELDALIAS-syslog_dst_address lines.
Doesn't that precedence only apply to configuration items that appear in multiple locations, and thus need to be overridden in some defined order?
That is, even if app config is higher priority here, there's no TRANSFORMS or TRANSFORMS-* items to process app-wise, thus we get the debug output showing that the active config items are TRANSFORMS = syslog-host and TRANSFORMS-mysystem = mysystem-nullqueue.
Why not do you write props.conf with nullqueue under etc/apps
?
Doesn't that precedence only apply to configuration items that appear in multiple locations, and thus need to be overridden in some defined order?
I don't know how it works. I hope someone answers.
Restarting after moving/duplicating the settings into etc/apps/search/local/ files still doesn't filter out any of the syslog data.
how 's btool
output?
new syslog message doesn't filter out?
The btool output is the same as before, but the "TRANSFORMS-mysystem = mysystem-nullqueue" line is showing as coming from the search files instead of the etc/system/local files, same with the transforms.conf header/dest_key/format/regex lines. All the unwanted syslog messages are still making it through to the index.
btool
output:
/opt/splunk/etc/apps/search/local/props.conf [syslog]
/opt/splunk/etc/apps/search/local/props.conf TRANSFORMS-mysystem = mysystem-nullqueue
These have to be this.
Current config after restart. The unwanted new messages are still being indexed. Is Splunk just not parsing these events for some reason? Is there any way to check and see if this syslog traffic is actually going through the right internal Splunk queues?
splunk cmd btool transforms list --debug :
/opt/splunk/etc/apps/search/local/transforms.conf [mysystem-nullqueue]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/transforms.conf DEST_KEY = queue
/opt/splunk/etc/apps/search/local/transforms.conf FORMAT = nullQueue
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/apps/search/local/transforms.conf REGEX = Name="support\.(ice|rest|dns|h323|bfcp|sip)"
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf WRITE_META = False
splunk cmd btool props list --debug :
/opt/splunk/etc/apps/search/local/props.conf [syslog]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/apps/search/local/props.conf EXTRACT-mysystem-syslog-apache = apache2.\d+.: (?<srcip>\S+).*?\"(?<method>\S+) (?<url>[^ ?]+)\?*(?<query>\S*) \S+\" \S+ (?<respcode>\d+) (?<respbytes>\S+) (?<resptime>\d+)
/opt/splunk/etc/apps/search/local/props.conf FIELDALIAS-syslog_dst_address = Dst_address ASNEW dest Dst_port ASNEW dest_port Src_address ASNEW src Src_port ASNEW src_port
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 32
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf REPORT-syslog = syslog-extractions
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf TIME_FORMAT = %b %d %H:%M:%S
/opt/splunk/etc/system/default/props.conf TRANSFORMS = syslog-host
/opt/splunk/etc/apps/search/local/props.conf TRANSFORMS-mysystem = mysystem-nullqueue
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf category = Operating System
/opt/splunk/etc/system/default/props.conf description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 3
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf pulldown_type = true
/opt/splunk/etc/system/default/props.conf sourcetype =
before: REGEX = Name=\"support\.(ice|bfcp|sip|rest|h323|dns)
now: REGEX = Name="support\.(ice|rest|dns|h323|bfcp|sip)"
simply:
REGEX = support\.(ice|rest|dns|h323|bfcp|sip)
How about this?
When REGEX matches the part of event , The event is null.
I've tried that, and replacing the regex string with just "REGEX = bfcp" or "REGEX = .bfcp." to try to eliminate the single set of events, but no matter what they keep being indexed.
My assumption is that the forwarder is actually a heavy forwarder so Splunk will not reparse the incoming data. Unfortunately the indexer is not under my control so I'm not sure how to find out what kind of data it's giving me.
indexer and HF need same props.conf(null queue.)
https://answers.splunk.com/answers/458237/setting-up-propsconf-at-the-heavy-forwarders.html
So if i don't control the forwarder and can't make changes on it, there's no way to have the indexer filter out these events?
sorry, I don't know.