Deployment Architecture

Is forwardedindex.. only applicable for TCP connection?

cemiam
Path Finder

Hi,

I am using heavy forwarder to forward syslogs to a 3rd party syslog aggregator. I am trying to filter some of the forwarded events on heavy forwarder and noticed that it is already sending audit logs even if I blacklist all internal indexes.

According to below outputs.conf it should not forward audit logs. After checking outputs.conf guide I have noticed forwardedindex..whitelist = and forwardedindex..blacklist = are only applicable under the global [tcpout] stanza. As it is an TCP based stanza I think it is not possible to filter UDP events. I need to make some additional regex-based filterings but I don't think it will not be possible. Is there any way to do this?

outputs.conf

[tcpout]
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
forwardedindex.filter.disable = false

[syslog]
defaultGroup = syslogGroup
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
forwardedindex.filter.disable = false

[syslog:syslogGroup]
server = 10.xx.x.xxx:514
sendCookedData = false

[syslog-server://10.xx.x.xxx:514]

[tcpout]
defaultGroup =
indexAndForward = 1

Tags (1)
0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Are you sure other outputs are not overriding your blacklist settings?

Try this to see if another app is overriding:

./splunk btool outputs list —debug

I’m not sure why you have indexAndForward enabled, you didn’t mention keeping the data on the heavy forwarder so I’m not sure if you want to do that or not.

As for applying your transformations to UDP data, yes you can do that.

See this documentation:
http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Routeandfilterdatad

Note you will be using _SYSLOG_ROUTING instead of _TCP_ROUTING. There’s also _HTTP_ROUTING according to the link.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Are you sure other outputs are not overriding your blacklist settings?

Try this to see if another app is overriding:

./splunk btool outputs list —debug

I’m not sure why you have indexAndForward enabled, you didn’t mention keeping the data on the heavy forwarder so I’m not sure if you want to do that or not.

As for applying your transformations to UDP data, yes you can do that.

See this documentation:
http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Routeandfilterdatad

Note you will be using _SYSLOG_ROUTING instead of _TCP_ROUTING. There’s also _HTTP_ROUTING according to the link.

cemiam
Path Finder

Thanks for the reply. I have enabled indexandForward just to troubleshooting. I have disabled it. You can find the btool commands output below. I couldn't find any overriding.

/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf [syslog]
/opt/splunk/etc/system/local/outputs.conf defaultGroup = syslogGroup
/opt/splunk/etc/system/local/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/local/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/system/local/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/system/local/outputs.conf [syslog-server://10.xx.x.xxx:514]
/opt/splunk/etc/system/local/outputs.conf [syslog:syslogGroup]
/opt/splunk/etc/system/local/outputs.conf server = 10.xx.x.xxx:514
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunk/etc/system/default/outputs.conf compressed = false
/opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunk/etc/system/local/outputs.conf defaultGroup =
/opt/splunk/etc/system/default/outputs.conf disabled = false
/opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunk/etc/system/local/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunk/etc/system/local/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunk/etc/apps/SplunkForwarder/local/outputs.conf forwardedindex.2.blacklist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_telemetry)
/opt/splunk/etc/system/local/outputs.conf forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunk/etc/system/local/outputs.conf indexAndForward = 0
/opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunk/etc/system/local/outputs.conf maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunk/etc/system/local/outputs.conf sendCookedData = false
/opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf useACK = false
/opt/splunk/etc/system/default/outputs.conf writeTimeout = 300
[root@heavyforwarder bin]#

For using transforms.conf I think I should index data. This is not something I should do. I have enabled indexandForward just for troubleshooting purposes.

0 Karma

cemiam
Path Finder

Somehow it doesn't allow me to upload inputs.conf output

0 Karma

cemiam
Path Finder

I am not sure what is wrong with the post but although I have enough characters left it doesn't allow me to upload inputs.con btool command output. You can find the my inputs.conf output under /local directory.

inputs.conf

[default]
host = heavyforwarder.dataserv.local

[udp://515]
sourcetype = syslog
disabled = 0

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sorry I meant props.conf and transforms.conf related to this input.

cemiam
Path Finder

I have tried to send btool outputs but the character limitation doesn't allow me to do it. I have re-checked the props.conf and transforms.conf. Both are on the default. As I am planning send this data directly to a 3rd party I didn't make any configuration.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Ok, so you haven’t done what the documentation says to do. See if this document is easier to follow:

http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Forwarddatatothird-partysystemsd

cemiam
Path Finder

Hi,

I have just followed this document and now I am able to filter specific event with the help of regex. The problem is I am still having trouble to blacklisting audit logs. As they are not in common format it is hard to filter audit logs with regex.

jkat54
SplunkTrust
SplunkTrust

Did you try a regex that matches the index name?

0 Karma

cemiam
Path Finder

According guide regex helps to filter data but I think it can be filtered via hostname on props.conf.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yeah you can do that.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you post the inputs.conf too?

0 Karma
Get Updates on the Splunk Community!

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...