Inconsistent source assignment using transforms.conf (MetaData:Source) with HEC JSON events

[luispulido]

Hello everyone,

I'm facing an inconsistent parsing issue while ingesting JSON events through HEC and I'd appreciate any guidance.

Goal

I'm trying to:

• Change the incoming sourcetype (currently arriving as json").
• Dynamically assign the source value based on the value of evt.name inside each event.

The sourcetype assignment works perfectly, but the dynamic source assignment only works for approximately 80-90% of the events. The remaining 10-20% keep the original source instead of being updated.

Current configuration

transforms.conf

[set_source_from_evt]
REGEX = .+\"evt\"\s*:\s*\{[\s\S]*?\"name\"\s*:\s*\"([^\"]+)\".+
FORMAT = source::$1
DEST_KEY = MetaData:Source

[set_datadog_sourcetype]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::datadog:audit

props.conf

[json"]
TRANSFORMS-fix_sourcetype = set_datadog_sourcetype, set_source_from_evt

I also tried applying the transform after changing the sourcetype:

[datadog:audit]
TRANSFORMS-set_source = set_source_from_evt

but the behavior remained exactly the same.

Troubleshooting already performed

• The regular expression has been tested with multiple online regex tools and matches 100% of the events, including those that fail during indexing.
• I also tested the same regex in Splunk Search (using rex) against indexed events, and it successfully extracts the value from every event.
• The issue is not tied to a specific value of evt.name; it appears to happen randomly.
• Sometimes an event is parsed correctly, while another event with the same structure is not.

For example, the following fragment should produce:

source = Synthetics Monitoring

because it contains:

"tags":["public_id:ec8-knz-4bv","version:4"],"evt":{"name":"Synthetics Monitoring","actor":{"type":"USER"}}},"message":

However, some events containing this structure still keep the original source.

Question

Has anyone experienced inconsistent behavior when setting MetaData:Source using a transform during HEC ingestion?

Is there any known limitation regarding:

• MetaData:Source transforms on HEC data?
• The use of [\s\S]*? in index-time regex?
• Regex evaluation at index time versus search time?
• Event parsing/truncation/order of processing that could explain why identical events are sometimes matched and sometimes not?

Any ideas or suggestions would be greatly appreciated.

Thanks!

[masonreed11]

You should check whether the events that fail are being truncated or parsed differently before the transform runs. Even if the regex works, rex, index-time transforms only see the raw event at ingestion, so settings like TRUNCATE, LINE_BREAKER, or INDEXED_EXTRACTIONS can affect the result.

Also enable parsing debug logs to confirm whether set_source_from_evt is actually matching those events or being skipped. If the same event sometimes works and sometimes doesn't, that points more toward the ingestion pipeline than the regex itself. If everything looks correct, it may be worth opening a Splunk support case, as inconsistent MetaData:Source transforms during HEC ingestion aren't expected.

[livehybrid · Accepted solution]

Hi @luispulido 

The inconsistency could be caused by the default LOOKAHEAD limit in transforms.conf. By default, Splunk only searches the first 4096 bytes of an event during index-time regex matching. If the "evt" block or "name" key is located beyond this limit in larger JSON payloads, the transform fails silently and retains the original source.

Add the following to your transforms stanza:

LOOKAHEAD=10000
# Or higher if required

Check out https://docs.splunk.com/Documentation/Splunk/8.2.12/Admin/Transformsconf#:~:text=LOOKAHEAD%20%3D%20%...

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful

• Marking it as the solution if it resolved your issue

• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.