Getting Data In

How can I use CLONE_SOURCETYPE to send a cloned modified event to a 3rd party without indexing the cloned event?

rphillips_splk
Splunk Employee
Splunk Employee

How can I use the CLONE_SOURCETYPE feature to clone an event that I need to modify and send to a 3rd party without indexing the cloned event as well? The intent is to index the original event and send a cloned (modified) version of the original event to a 3rd party.

0 Karma
1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

One solution I have tested successfully is by using a Heavy Forwarder (full Splunk Enterprise instance, not Universal Forwarder) to do the clone_sourcetype before sending the events to the indexer.

All configurations below are applied to the heavy forwarder

1) create your tcpout stanza for forwarding data to your indexers
outputs.conf

[tcpout:clustered_indexers]
server = 10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

2) set _TCP_ROUTING in inputs.conf under the default stanza

inputs.conf

[default]
host = splunk-hfwd01
_TCP_ROUTING = clustered_indexers

3) since _TCP_ROUTING is defined in [default] it will get applied to all inputs

4) here is the input which we will clone events from
inputs.conf
[monitor:///opt/splunk/logs/test.log]
_TCP_ROUTING = clustered_indexers
disabled = 0
host = splunk-hfwd01
index = main
sourcetype = original

5) configure props stanza for the source and point it to transforms to apply CLONE_SOURCETYPE to this data
props.conf
[source::.../test.log]
TRANSFORMS-clone = clone_sourcetype

6) configure the transforms to assign a new sourcetype called cloned to the cloned events.
transforms.conf
[clone_sourcetype]
CLONE_SOURCETYPE = cloned
REGEX = .

7) configure the cloned sourcetype stanza in props where you will modify the cloned events ie: SEDCMD, LineBreaking etc. Also assign two transforms to these events to route them to the syslog output and to NOT route them out the TCP output. This will prevent a copy of the cloned event from being sent to your indexers as defined in _TCP_ROUTING 'clustered_indexers'

props.conf
[cloned]

apply any props changes appropriate to your data for the new cloned sourcetype

SEDCMD-custom = s/[\n\r\t]/ /g
BREAK_ONLY_BEFORE = ((.+)\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+([aApPmM]{2}))
TRANSFORMS-output = cloned_syslog,cloned_noTCP_routing

😎 configure the transforms for syslog routing and NO TCP routing. Here we route to a bogus destination for _TCP_ROUTING which does not exist since we only want to send these cloned events to the syslog output processor.

transforms.conf
[cloned_syslog]
DEST_KEY = _SYSLOG_ROUTING
FORMAT = send_syslog_to_3rdParty
REGEX = .

[cloned_noTCP_routing]
DEST_KEY = _TCP_ROUTING
FORMAT = bogus
REGEX = .

9) configure outputs
outputs.conf

[syslog:send_syslog_to_3rdParty]
priority = <13>
server = 10.10.10.25:514
timestampformat = <%b %e %H:%M:%S>
type = udp

[tcpout]
defaultGroup = bogus

10) the result of such configurations should send the original event to your index cluster and a cloned copy of the event to the 3rd party syslog receiver.

Caution: Sending data to a single receiver can cause queues on the HF to block if that receiver goes down.

View solution in original post

BDein
Explorer

Hi @rphillips_splk,
While I know this is some time ago, I still find this very interesting!
You used 2 different routing types here - so I need to ask if this also could be applied on 2 TCP (different) connections, so the cloned version could also be send as TCP and not syslog?

Moreover - I'm in the situation, there will be an additional HF where you show the "syslog receiver" above, and the actual indexer - so basically route like this:
UF -> HF (clone) -> IDX
                       |-> HF -> IDX

Can this be done as smooth as above?
If so, how?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

One solution I have tested successfully is by using a Heavy Forwarder (full Splunk Enterprise instance, not Universal Forwarder) to do the clone_sourcetype before sending the events to the indexer.

All configurations below are applied to the heavy forwarder

1) create your tcpout stanza for forwarding data to your indexers
outputs.conf

[tcpout:clustered_indexers]
server = 10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

2) set _TCP_ROUTING in inputs.conf under the default stanza

inputs.conf

[default]
host = splunk-hfwd01
_TCP_ROUTING = clustered_indexers

3) since _TCP_ROUTING is defined in [default] it will get applied to all inputs

4) here is the input which we will clone events from
inputs.conf
[monitor:///opt/splunk/logs/test.log]
_TCP_ROUTING = clustered_indexers
disabled = 0
host = splunk-hfwd01
index = main
sourcetype = original

5) configure props stanza for the source and point it to transforms to apply CLONE_SOURCETYPE to this data
props.conf
[source::.../test.log]
TRANSFORMS-clone = clone_sourcetype

6) configure the transforms to assign a new sourcetype called cloned to the cloned events.
transforms.conf
[clone_sourcetype]
CLONE_SOURCETYPE = cloned
REGEX = .

7) configure the cloned sourcetype stanza in props where you will modify the cloned events ie: SEDCMD, LineBreaking etc. Also assign two transforms to these events to route them to the syslog output and to NOT route them out the TCP output. This will prevent a copy of the cloned event from being sent to your indexers as defined in _TCP_ROUTING 'clustered_indexers'

props.conf
[cloned]

apply any props changes appropriate to your data for the new cloned sourcetype

SEDCMD-custom = s/[\n\r\t]/ /g
BREAK_ONLY_BEFORE = ((.+)\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+([aApPmM]{2}))
TRANSFORMS-output = cloned_syslog,cloned_noTCP_routing

😎 configure the transforms for syslog routing and NO TCP routing. Here we route to a bogus destination for _TCP_ROUTING which does not exist since we only want to send these cloned events to the syslog output processor.

transforms.conf
[cloned_syslog]
DEST_KEY = _SYSLOG_ROUTING
FORMAT = send_syslog_to_3rdParty
REGEX = .

[cloned_noTCP_routing]
DEST_KEY = _TCP_ROUTING
FORMAT = bogus
REGEX = .

9) configure outputs
outputs.conf

[syslog:send_syslog_to_3rdParty]
priority = <13>
server = 10.10.10.25:514
timestampformat = <%b %e %H:%M:%S>
type = udp

[tcpout]
defaultGroup = bogus

10) the result of such configurations should send the original event to your index cluster and a cloned copy of the event to the 3rd party syslog receiver.

Caution: Sending data to a single receiver can cause queues on the HF to block if that receiver goes down.

raventura
Observer

Hi @rphillips_splk , is there a way to specify a certain source or host from the selected sourcetype to be cloned? thanks!

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

This was tested on the following versions:

HF: 6.4.1

Indexers: 6.6.1

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

alt text

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...