Getting Data In

Forwarding based on both host and sourcetype

joelshprentz
Path Finder

We would like to forward some data from our Splunk indexer to various third party destinations based on the host and sourcetype of the data. The Splunk documentation explains how to configure forwarding for a particular host, sourcetype, or source, but not for a combination such as host and source. How can it be done?

For example, suppose we want to forward logs from two hosts, host1 and host2. We only want to forward sourcetypes log4j and syslog. This table shows the desired host:port destinations for various combinations of log host and sourcetype:

 log4jsyslog
host1dest1:4001dest2:4002
host2dest3:5001dest4:5002

 

The documentation for routing and filtering data explains how the props.conf stanza names specify the filtering:

  • sourcetype — the source type of an event
  • host::host — where host is the host for an event
  • source::source — where source is the source for an event

How can a props.conf stanza name combine the notation for sourcetype and host?

1 Solution

joelshprentz
Path Finder

The excellent response from gkanapathy needed a few changes to work in our Splunk 4.3.1 installation. The hardest part was discovering the SOURCE_KEY for hosts; it is MetaData:Host.

Here are the resulting configuration stanzas ...

outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = nothing

[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1

[tcpout:dest-host1-log4j]
server = dest1:6001
sendCookedData = false

[tcpout:dest-host1-syslog]
server = dest1:6002
sendCookedData = false

[tcpout:dest-host2-log4j]
server = dest1:6003
sendCookedData = false

[tcpout:dest-host2-syslog]
server = dest1:6004
sendCookedData = false

Changes:

  • Send logs via TCP instead of syslog.
  • Force raw format for our third-party destination.
  • Index locally in addition to forwarding.
  • Forward only logs for the designated hosts and source types.
  • Worked around for bug SPL-50576 in Splunk 4.3.1 and 4.3.2. See this answer.

props.conf

[log4j]
TRANSFORMS-transform-host1-log4j = route-host1-log4j
TRANSFORMS-transform-host2-log4j = route-host2-log4j

[syslog]
TRANSFORMS-transform-host1-syslog = route-host1-syslog
TRANSFORMS-transform-host2-syslog = route-host2-syslog

Unchanged.

transforms.conf

[route-host1-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-log4j

[route-host1-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-syslog

[route-host2-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-log4j

[route-host2-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-syslog

Changes:

  • Send logs via TCP instead of syslog.
  • Use the documented SOURCE_KEY for the host.

View solution in original post

joelshprentz
Path Finder

The excellent response from gkanapathy needed a few changes to work in our Splunk 4.3.1 installation. The hardest part was discovering the SOURCE_KEY for hosts; it is MetaData:Host.

Here are the resulting configuration stanzas ...

outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = nothing

[tcpout:nothing]
disabled = false
server = falsefoo.bar.com:9998
dropEventsOnQueueFull = 1

[tcpout:dest-host1-log4j]
server = dest1:6001
sendCookedData = false

[tcpout:dest-host1-syslog]
server = dest1:6002
sendCookedData = false

[tcpout:dest-host2-log4j]
server = dest1:6003
sendCookedData = false

[tcpout:dest-host2-syslog]
server = dest1:6004
sendCookedData = false

Changes:

  • Send logs via TCP instead of syslog.
  • Force raw format for our third-party destination.
  • Index locally in addition to forwarding.
  • Forward only logs for the designated hosts and source types.
  • Worked around for bug SPL-50576 in Splunk 4.3.1 and 4.3.2. See this answer.

props.conf

[log4j]
TRANSFORMS-transform-host1-log4j = route-host1-log4j
TRANSFORMS-transform-host2-log4j = route-host2-log4j

[syslog]
TRANSFORMS-transform-host1-syslog = route-host1-syslog
TRANSFORMS-transform-host2-syslog = route-host2-syslog

Unchanged.

transforms.conf

[route-host1-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-log4j

[route-host1-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host1
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host1-syslog

[route-host2-log4j]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-log4j

[route-host2-syslog]
SOURCE_KEY = MetaData:Host
REGEX = host2
DEST_KEY = _TCP_ROUTING
FORMAT = dest-host2-syslog

Changes:

  • Send logs via TCP instead of syslog.
  • Use the documented SOURCE_KEY for the host.

gkanapathy
Splunk Employee
Splunk Employee

It can't. The only way I can think is to do multiple transforms.

outputs.conf

[syslog:dest1]
server = dest1:4001
[syslog:dest3]
server = dest3:5001

props.conf:

[log4j]
TRANSFORMS-route-host1 = rt-log4j-host1
TRANSFORMS-route-host2 = rt-log4j-host2

transforms.conf:

[rt-log4j-host1]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host1
FORMAT = dest1

[rt-log4j-host2]
SOURCE_KEY = host
DEST_KEY = _SYSLOG_ROUTING
REGEX = host2
FORMAT = dest3

And similar for the syslog sourcetype.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...