Splunk Enterprise

Is CLONE_SOURCETYPE works properly?

NoSpaces
Contributor

For some reason, I needed to share some data from an index with a different set of permissions.
After a bit of research, I found that the CLONE_SOURCETYPE option could help me with this stuff.
I created the required settings in props.conf and transforms.conf, and then pushed them to the IDXC layer.
At first glance, everything seemed fine, but then I discovered that CLONE_SOURCETYPE clones all events from the original sourcetype and redirects only a few to the new one.
Is that the intended behavior, or did I make serious mistakes in the configuration?
I expected to see only the events matching the REGEX in the original index.

 

props.conf

[vsi_file_esxi-syslog]
LINE_BREAKER = (\n)
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = \d{1,3}
TIME_PREFIX = ^<\d{1,3}>
TRANSFORMS-remove_trash = vsi_file_esxi-syslog_rt0, vsi_file_esxi-syslog_ke0
TRANSFORMS-route_events = general_file_esxi-syslog_re0
transforms.conf

[general_file_esxi-syslog_re0]
CLONE_SOURCETYPE = general_re_esxi-syslog
REGEX = FIREWALL-PKTLOG:
DEST_KEY = _MetaData:Index
FORMAT = general
WRITE_META = true

 

 

Labels (1)
0 Karma
1 Solution

kiran_panchavat
Champion

@NoSpaces 

Yes, the behavior you're observing with CLONE_SOURCETYPE is expected. When you use CLONE_SOURCETYPE in Splunk, it creates a duplicate of every event that matches the props.conf stanza, regardless of the REGEX specified in the corresponding transforms.conf stanza. The REGEX is applied to the cloned event, not to determine whether an event should be cloned in the first place. This means that all events are cloned, and then the REGEX is used to modify or route the cloned events as specified.

https://community.splunk.com/t5/Getting-Data-In/Priority-precedence-in-props-conf/m-p/669047 

https://community.splunk.com/t5/Getting-Data-In/Can-I-use-CLONE-SOURCETYPE-to-send-events-to-multipl... 

To clone only the events matching the REGEX to the new sourcetype and redirect them to the general index, while keeping all original events in the original index under the original sourcetype, you need to filter events before cloning. Unfortunately, Splunk’s CLONE_SOURCETYPE doesn’t natively support filtering during cloning.
 
You can use two transforms: one to filter out events that don’t match the REGEX and send them to nullQueue (discarding them from cloning), and another to clone and redirect the matching events. 
 
  • Events matching FIREWALL-PKTLOG: are cloned and routed to the general index.
  • The same matching events are dropped from the original index using nullQueue.
Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

CLONE_SOURCETYPE works on all events on which it is fired regardless of the REGEX value. In other words - you cannot limit its scope. If you assign a transform with CLONE_SOURCETYPE to a sourcetype, source or host, it will clone your event without any filtering.

And yes, the docs on CLONE_SOURCETYPE are a bit misleading and confusing.

0 Karma

kiran_panchavat
Champion

@NoSpaces 

Yes, the behavior you're observing with CLONE_SOURCETYPE is expected. When you use CLONE_SOURCETYPE in Splunk, it creates a duplicate of every event that matches the props.conf stanza, regardless of the REGEX specified in the corresponding transforms.conf stanza. The REGEX is applied to the cloned event, not to determine whether an event should be cloned in the first place. This means that all events are cloned, and then the REGEX is used to modify or route the cloned events as specified.

https://community.splunk.com/t5/Getting-Data-In/Priority-precedence-in-props-conf/m-p/669047 

https://community.splunk.com/t5/Getting-Data-In/Can-I-use-CLONE-SOURCETYPE-to-send-events-to-multipl... 

To clone only the events matching the REGEX to the new sourcetype and redirect them to the general index, while keeping all original events in the original index under the original sourcetype, you need to filter events before cloning. Unfortunately, Splunk’s CLONE_SOURCETYPE doesn’t natively support filtering during cloning.
 
You can use two transforms: one to filter out events that don’t match the REGEX and send them to nullQueue (discarding them from cloning), and another to clone and redirect the matching events. 
 
  • Events matching FIREWALL-PKTLOG: are cloned and routed to the general index.
  • The same matching events are dropped from the original index using nullQueue.
Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...