We would like to forward some data from our Splunk indexer to various third party destinations based on the host and sourcetype of the data. The Splunk documentation explains how to configure forwarding for a particular host, sourcetype, or source, but not for a combination such as host and source. How can it be done?
For example, suppose we want to forward logs from two hosts, host1 and host2. We only want to forward sourcetypes log4j and syslog. This table shows the desired host:port destinations for various combinations of log host and sourcetype:
log4j | syslog | |
host1 | dest1:4001 | dest2:4002 |
host2 | dest3:5001 | dest4:5002 |
The documentation for routing and filtering data explains how the props.conf stanza names specify the filtering:
How can a props.conf stanza name combine the notation for sourcetype and host?
The excellent response from gkanapathy needed a few changes to work in our Splunk 4.3.1 installation. The hardest part was discovering the SOURCE_KEY for hosts; it is MetaData:Host
.
Here are the resulting configuration stanzas ...
outputs.conf
[tcpout]
indexAndForward = true
defaultGroup = nothing
[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1
[tcpout:dest-host1-log4j]
server = dest1:6001
sendCookedData = false
[tcpout:dest-host1-syslog]
server = dest1:6002
sendCookedData = false
[tcpout:dest-host2-log4j]
server = dest1:6003
sendCookedData = false
[tcpout:dest-host2-syslog]
server = dest1:6004
sendCookedData = false
Changes:
props.conf
[log4j]
TRANSFORMS-transform-host1-log4j = route-host1-log4j
TRANSFORMS-transform-host2-log4j = route-host2-log4j
[syslog]
TRANSFORMS-transform-host1-syslog = route-host1-syslog
TRANSFORMS-transform-host2-syslog = route-host2-syslog
Unchanged.
transforms.conf
[route-host1-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-log4j
[route-host1-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-syslog
[route-host2-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-log4j
[route-host2-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-syslog
Changes:
The excellent response from gkanapathy needed a few changes to work in our Splunk 4.3.1 installation. The hardest part was discovering the SOURCE_KEY for hosts; it is MetaData:Host
.
Here are the resulting configuration stanzas ...
outputs.conf
[tcpout]
indexAndForward = true
defaultGroup = nothing
[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1
[tcpout:dest-host1-log4j]
server = dest1:6001
sendCookedData = false
[tcpout:dest-host1-syslog]
server = dest1:6002
sendCookedData = false
[tcpout:dest-host2-log4j]
server = dest1:6003
sendCookedData = false
[tcpout:dest-host2-syslog]
server = dest1:6004
sendCookedData = false
Changes:
props.conf
[log4j]
TRANSFORMS-transform-host1-log4j = route-host1-log4j
TRANSFORMS-transform-host2-log4j = route-host2-log4j
[syslog]
TRANSFORMS-transform-host1-syslog = route-host1-syslog
TRANSFORMS-transform-host2-syslog = route-host2-syslog
Unchanged.
transforms.conf
[route-host1-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-log4j
[route-host1-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-syslog
[route-host2-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-log4j
[route-host2-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-syslog
Changes:
It can't. The only way I can think is to do multiple transforms.
outputs.conf
[syslog:dest1]
server = dest1:4001
[syslog:dest3]
server = dest3:5001
props.conf:
[log4j]
TRANSFORMS-route-host1 = rt-log4j-host1
TRANSFORMS-route-host2 = rt-log4j-host2
transforms.conf:
[rt-log4j-host1]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host1
FORMAT = dest1
[rt-log4j-host2]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host2
FORMAT = dest3
And similar for the syslog
sourcetype.