topic Re: Is forwardedindex.. only applicable for TCP connection? in Deployment Architecture

Is forwardedindex.. only applicable for TCP connection?

cemiam — Tue, 29 Sep 2020 18:57:56 GMT

Hi,

I am using heavy forwarder to forward syslogs to a 3rd party syslog aggregator. I am trying to filter some of the forwarded events on heavy forwarder and noticed that it is already sending audit logs even if I blacklist all internal indexes.

According to below outputs.conf it should not forward audit logs. After checking outputs.conf guide I have noticed forwardedindex..whitelist = and forwardedindex..blacklist = are only applicable under the global [tcpout] stanza. As it is an TCP based stanza I think it is not possible to filter UDP events. I need to make some additional regex-based filterings but I don't think it will not be possible. Is there any way to do this?

outputs.conf

[tcpout]
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
forwardedindex.filter.disable = false

[syslog]
defaultGroup = syslogGroup
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
forwardedindex.filter.disable = false

[syslog:syslogGroup]
server = 10.xx.x.xxx:514
sendCookedData = false

[syslog-server://10.xx.x.xxx:514]

[tcpout]
defaultGroup =
indexAndForward = 1

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Tue, 29 Sep 2020 18:55:29 GMT

Are you sure other outputs are not overriding your blacklist settings?

Try this to see if another app is overriding:

./splunk btool outputs list —debug

I’m not sure why you have indexAndForward enabled, you didn’t mention keeping the data on the heavy forwarder so I’m not sure if you want to do that or not.

As for applying your transformations to UDP data, yes you can do that.

See this documentation:
http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Routeandfilterdatad

Note you will be using _SYSLOG_ROUTING instead of _TCP_ROUTING. There’s also _HTTP_ROUTING according to the link.

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Tue, 29 Sep 2020 18:57:59 GMT

Thanks for the reply. I have enabled indexandForward just to troubleshooting. I have disabled it. You can find the btool commands output below. I couldn't find any overriding.

/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf [syslog]
/opt/splunk/etc/system/local/outputs.conf defaultGroup = syslogGroup
/opt/splunk/etc/system/local/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/local/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/system/local/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/local/outputs.conf [syslog-server://10.xx.x.xxx:514]
/opt/splunk/etc/system/local/outputs.conf [syslog:syslogGroup]
/opt/splunk/etc/system/local/outputs.conf server = 10.xx.x.xxx:514
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf defaultGroup =
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/local/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/local/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/system/local/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/local/outputs.conf indexAndForward = 0
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/local/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
[root@heavyforwarder bin]#

For using transforms.conf I think I should index data. This is not something I should do. I have enabled indexandForward just for troubleshooting purposes.

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Tue, 10 Apr 2018 11:33:19 GMT

Can you post the inputs.conf too?

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Tue, 10 Apr 2018 12:06:00 GMT

Somehow it doesn't allow me to upload inputs.conf output

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Tue, 10 Apr 2018 12:10:19 GMT

I am not sure what is wrong with the post but although I have enough characters left it doesn't allow me to upload inputs.con btool command output. You can find the my inputs.conf output under /local directory.

inputs.conf

[default]
host = heavyforwarder.dataserv.local

[udp://515]
sourcetype = syslog
disabled = 0

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Tue, 10 Apr 2018 13:43:33 GMT

Sorry I meant props.conf and transforms.conf related to this input.

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Wed, 11 Apr 2018 07:34:34 GMT

I have tried to send btool outputs but the character limitation doesn't allow me to do it. I have re-checked the props.conf and transforms.conf. Both are on the default. As I am planning send this data directly to a 3rd party I didn't make any configuration.

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Wed, 11 Apr 2018 11:16:22 GMT

Ok, so you haven’t done what the documentation says to do. See if this document is easier to follow:

http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Forwarddatatothird-partysystemsd

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Wed, 11 Apr 2018 11:29:02 GMT

Hi,

I have just followed this document and now I am able to filter specific event with the help of regex. The problem is I am still having trouble to blacklisting audit logs. As they are not in common format it is hard to filter audit logs with regex.

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Thu, 12 Apr 2018 22:17:21 GMT

Did you try a regex that matches the index name?

Re: Is forwardedindex.. only applicable for TCP connection?

cemiam — Fri, 13 Apr 2018 07:49:51 GMT

According guide regex helps to filter data but I think it can be filtered via hostname on props.conf.

Re: Is forwardedindex.. only applicable for TCP connection?

jkat54 — Fri, 13 Apr 2018 07:53:50 GMT

Yeah you can do that.