Index redirection using props.conf and transforms.conf not working (Splunk 9.4.6)

[Cybers1]

Hi everyone, I need to open a case here hoping someone can help us. We need to redirect logs from specific sourcetypes to different target indexes. To achieve this, we configured index-time transformations using props.conf and transforms.conf. We applied the configuration both on the main indexer and on the Heavy Forwarder. Below are our configurations.

## transforms.conf
[redirect_source_k8_accesslog_coll]
REGEX = .*\[ACCESSLOG_COLL\].*
FORMAT = accesslog_coll
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
WRITE_META = true 

[redirect_source_k8_accesslog]
REGEX = .*\[ACCESSLOG\].*
FORMAT = accesslog
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
WRITE_META = true

## props.conf
[kube:container:*]
TRANSFORMS-k8_accesslog_and_accesslog_coll_redirect = redirect_source_k8_accesslog_coll, redirect_source_k8_accesslog 

However, the redirection is still not working. We are running Splunk Enterprise 9.4.6. Could you please help us understand: If we are missing something in the configuration? If the configuration placement (Indexer vs Heavy Forwarder) could be the issue? If there are better or recommended approaches in newer Splunk versions to redirect events to specific indexes? Any guidance would be greatly appreciated. Thank you in advance!

[kknairr]

@Cybers1 Few things here,

• Make sure the indexes are created and configured correctly in indexes.conf which is referenced in your config files
• Props stanza matching, make sure the sourcetype matches the format defined.
• Regarding placement of configurations. If both HF and Indexer have configs, it's okay to have it, but sometimes you can get conflicts due to mismatches. It's best to keep them only in HF as it could do the parsing and if you are keeping it in both, make sure those are identical.
• For regex pattern, can you try the below props and transforms conf file in dev testing:

transforms.conf

[redirect_source_k8_accesslog_coll]
REGEX = \[ACCESSLOG_COLL\]
FORMAT = accesslog_coll
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
WRITE_META = true

[redirect_source_k8_accesslog]
REGEX = \[ACCESSLOG\]
FORMAT = accesslog
SOURCE_KEY = _raw
DEST_KEY = _MetaData:Index
WRITE_META = true

props.conf

[kube:container:*]
TRANSFORMS-routing = redirect_source_k8_accesslog_coll, redirect_source_k8_accesslog

[livehybrid]

You cant have wildcards in props stanzas like that? The docs state

##########
# Wildcard sourcetypes - multiple sourcetypes that begin with the same string 
##########

[(?::){0}acme:*]
LOOKUP-acme = lookup acme_users user_id AS user_id OUTPUTNEW user_name AS
\ user_name FirstName AS FirstName LastName AS LastName

See also: https://community.splunk.com/t5/Getting-Data-In/Having-Difficulty-Understanding-Stanza-in-props-conf...

[PickleRick]

Yeah. "Directly" sourcetype-based stanza must match a single sourcetype literally.

But apparently the engine matching event to relevant props.conf settings has something about treating everything containing :: as either source or host-based setting so it applies the setting anyway using the regex matching method. Since the :: appears literally in the stanza but is completely ignored in matching, that works. But boy, it's such an ugly hack...

[livehybrid · Accepted solution]

Hi @Cybers1 

I believe the problem here is the wildcard in the sourcetype name, could you try updating your props stanza to:

[(?:::){0}kube:container:*]
TRANSFORMS-k8_accesslog_and_accesslog_coll_redirect = redirect_source_k8_accesslog_coll, redirect_source_k8_accesslog 

For more info check out https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/configuration-file-referenc...

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Cybers1]

Hi,

Thank you for your suggestion.

We already tried modifying the props stanza using a regex-based sourcetype match, specifically:

[(?::){0}kube:.*] TRANSFORMS-k8_accesslog_and_accesslog_coll_redirect = redirect_source_k8_accesslog_coll, redirect_source_k8_accesslog

However, even with this change, the index redirection is still not working.

Any additional suggestions would be greatly appreciated.

Thanks again for your support.