Getting Data In
Highlighted

How can I use CLONE_SOURCETYPE to send a cloned modified event to a 3rd party without indexing the cloned event?

Splunk Employee
Splunk Employee

How can I use the CLONE_SOURCETYPE feature to clone an event that I need to modify and send to a 3rd party without indexing the cloned event as well? The intent is to index the original event and send a cloned (modified) version of the original event to a 3rd party.

0 Karma
Highlighted

Re: How can I use CLONE_SOURCETYPE to send a cloned modified event to a 3rd party without indexing the cloned event?

Splunk Employee
Splunk Employee

One solution I have tested successfully is by using a Heavy Forwarder (full Splunk Enterprise instance, not Universal Forwarder) to do the clone_sourcetype before sending the events to the indexer.

All configurations below are applied to the heavy forwarder

1) create your tcpout stanza for forwarding data to your indexers
outputs.conf

[tcpout:clustered_indexers]
server = 10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

2) set TCPROUTING in inputs.conf under the default stanza

inputs.conf

[default]
host = splunk-hfwd01
TCPROUTING = clustered_indexers

3) since TCPROUTING is defined in [default] it will get applied to all inputs

4) here is the input which we will clone events from
inputs.conf
[monitor:///opt/splunk/logs/test.log]
TCPROUTING = clustered_indexers
disabled = 0
host = splunk-hfwd01
index = main
sourcetype = original

5) configure props stanza for the source and point it to transforms to apply CLONESOURCETYPE to this data
props.conf
[source::.../test.log]
TRANSFORMS-clone = clone
sourcetype

6) configure the transforms to assign a new sourcetype called cloned to the cloned events.
transforms.conf
[clonesourcetype]
`CLONE
SOURCETYPE` = cloned
REGEX = .

7) configure the cloned sourcetype stanza in props where you will modify the cloned events ie: SEDCMD, LineBreaking etc. Also assign two transforms to these events to route them to the syslog output and to NOT route them out the TCP output. This will prevent a copy of the cloned event from being sent to your indexers as defined in TCPROUTING 'clustered_indexers'

props.conf
[cloned]

apply any props changes appropriate to your data for the new cloned sourcetype

SEDCMD-custom = s/[\n\r\t]/ /g
BREAKONLYBEFORE = ((.+)\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+([aApPmM]{2}))
TRANSFORMS-output = clonedsyslog,clonednoTCP_routing

😎 configure the transforms for syslog routing and NO TCP routing. Here we route to a bogus destination for TCPROUTING which does not exist since we only want to send these cloned events to the syslog output processor.

transforms.conf
[clonedsyslog]
DEST
KEY = SYSLOGROUTING
FORMAT = sendsyslogto_3rdParty
REGEX = .

[clonednoTCProuting]
DESTKEY = _TCPROUTING
FORMAT = bogus
REGEX = .

9) configure outputs
outputs.conf

[syslog:sendsyslogto_3rdParty]
priority = <13>
server = 10.10.10.25:514
timestampformat = <%b %e %H:%M:%S>
type = udp

[tcpout]
defaultGroup = bogus

10) the result of such configurations should send the original event to your index cluster and a cloned copy of the event to the 3rd party syslog receiver.

Caution: Sending data to a single receiver can cause queues on the HF to block if that receiver goes down.

View solution in original post

Highlighted

Re: How can I use CLONE_SOURCETYPE to send a cloned modified event to a 3rd party without indexing the cloned event?

Splunk Employee
Splunk Employee
0 Karma
Highlighted

Re: How can I use CLONE_SOURCETYPE to send a cloned modified event to a 3rd party without indexing the cloned event?

Splunk Employee
Splunk Employee

This was tested on the following versions:

HF: 6.4.1

Indexers: 6.6.1

0 Karma