Override host and sourcetype value with event data

[JohnSmith123]

Hello everyone. I'm trying to set host and sourcetype values with event data. The result is that, the sourcetype is overridden as expected, while the host value is NOT.

By applying the following tranforms.conf and props.conf, I expect that

1. The sourcetype is overridden from default:log to mysp(which works as expected).
2. Then, for events with sourcetype mysp, override the host value with my event data using regex extraction(which didn't work).  

It's making me confused. Wondering why it didn't work out for host field. Hopefully someone would kindly help me out here. Thanks.

• transforms.conf

[sourcetype_transform]
SOURCE_KEY = _raw
REGEX = <my_regex>
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::mysp

[host_transform]
REGEX = <my_regex>
FORMAT = host::$1
DEST_KEY = MetaData:Host

•  props.conf

[default:log]
TRANSFORMS-force_sourcetype = sourcetype_transform

[mysp]
TRANSFORMS-force_host = host_transform

[PickleRick]

If I understand you correctly, you're rewriting sourcetype to mysp and then expect Splunk to apply transforms define for that sourcetype to the events further down the ingestion pipeline. It doesn't work that way (but it's a common expectation, I myself thought it did a few years ago).

Splunk decides at the beginning of the pipeline which settings apply to sourcetype/source/host triple and subsequent rewrites to those fields do not change it - the event goes through ingestion pipeline using the originally decided transforms.

The only way to "switch" to another sourcetype is to use CLONE_SOURCETYPE (but then you have to handle the original copy of the event as well).

[JohnSmith123]

Yes, this is exactly what I expected. Thank you for confirming the way it works.

[livehybrid · Accepted solution]

Hi @JohnSmith123 

I dont think you get two-bites of the props.conf cherry when changing a sourcetype name, instead you need to apply your host_transform to the "default:log" sourcetype rather than the new sourcetype name.

Try the following:

== props.conf == 
[default:log]
TRANSFORMS-force_sourcetype = sourcetype_transform
TRANSFORMS-force_host = host_transform

== transforms.conf ==
[sourcetype_transform]
SOURCE_KEY = _raw
REGEX = <my_regex>
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::mysp

[host_transform]
REGEX = <my_regex>
FORMAT = host::$1
DEST_KEY = MetaData:Host

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[JohnSmith123]

Hello @livehybrid . It works. Thanks a lot

[kiran_panchavat]

@JohnSmith123 

Ensure that your regex in host_transform correctly matches the part of the event data you want to extract as the host. You can test your regex separately to confirm it captures the desired value.

please provide:

1. The actual REGEX used in host_transform and sourcetype_transform.
2. A sample of the raw event data (_raw).
3. Details about where the configurations are deployed (e.g., heavy forwarder).