Forwarding based on both host and sourcetype

[joelshprentz]

We would like to forward some data from our Splunk indexer to various third party destinations based on the host and sourcetype of the data. The Splunk documentation explains how to configure forwarding for a particular host, sourcetype, or source, but not for a combination such as host and source. How can it be done?

For example, suppose we want to forward logs from two hosts, host1 and host2. We only want to forward sourcetypes log4j and syslog. This table shows the desired host:port destinations for various combinations of log host and sourcetype:

	log4j	syslog
host1	dest1:4001	dest2:4002
host2	dest3:5001	dest4:5002

The documentation for routing and filtering data explains how the props.conf stanza names specify the filtering:

• sourcetype — the source type of an event
• host::host — where host is the host for an event
• source::source — where source is the source for an event

How can a props.conf stanza name combine the notation for sourcetype and host?

[joelshprentz · Accepted solution]

The excellent response from gkanapathy needed a few changes to work in our Splunk 4.3.1 installation. The hardest part was discovering the SOURCE_KEY for hosts; it is MetaData:Host.

Here are the resulting configuration stanzas ...

outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = nothing

[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1

[tcpout:dest-host1-log4j]
server = dest1:6001
sendCookedData = false

[tcpout:dest-host1-syslog]
server = dest1:6002
sendCookedData = false

[tcpout:dest-host2-log4j]
server = dest1:6003
sendCookedData = false

[tcpout:dest-host2-syslog]
server = dest1:6004
sendCookedData = false

Changes:

• Send logs via TCP instead of syslog.
• Force raw format for our third-party destination.
• Index locally in addition to forwarding.
• Forward only logs for the designated hosts and source types.
• Worked around for bug SPL-50576 in Splunk 4.3.1 and 4.3.2. See this answer.

props.conf

[log4j]
TRANSFORMS-transform-host1-log4j = route-host1-log4j
TRANSFORMS-transform-host2-log4j = route-host2-log4j

[syslog]
TRANSFORMS-transform-host1-syslog = route-host1-syslog
TRANSFORMS-transform-host2-syslog = route-host2-syslog

Unchanged.

transforms.conf

[route-host1-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-log4j

[route-host1-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-syslog

[route-host2-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-log4j

[route-host2-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-syslog

Changes:

• Send logs via TCP instead of syslog.
• Use the documented SOURCE_KEY for the host.

[gkanapathy]

It can't. The only way I can think is to do multiple transforms.

outputs.conf

[syslog:dest1]
server = dest1:4001
[syslog:dest3]
server = dest3:5001

props.conf:

[log4j]
TRANSFORMS-route-host1 = rt-log4j-host1
TRANSFORMS-route-host2 = rt-log4j-host2

transforms.conf:

[rt-log4j-host1]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host1
FORMAT = dest1

[rt-log4j-host2]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host2
FORMAT = dest3

And similar for the syslog sourcetype.