Source attribute has to be a certain length?

[bgresty]

Hi, we've encountered some unusual behaviour when ingesting data and are at a loss as to what might be causing it. We have two presumably identical indexes ingesting identically structured messages from different regions via an HEC for each.

Index 1 has no issue, all messages are ingested no problem.

On Index 2, events only appear if the source attribute of the message is equal or greater than 14 characters long.

e.g.:

{
   other_data: ...
   source: 12345678901234
}

Any ideas?

[PrewinThomas]

@bgresty I strongly agree with what @livehybrid says.
Your 14 characters highlight, pointing towards a configuration that's unintentionally filtering or misinterpreting data based on this field's length which should be in your props/transforms conf.

[livehybrid]

Hi @bgresty 

This behavior is almost certainly caused by ingestion-time processing rules configured in props.conf and transforms.conf on the indexer(s) receiving data for Index 2.

Look for stanzas in props.conf that match the source type, host, or index of the data being sent to Index 2. These stanzas might contain TRANSFORMS, FILTER, or ROUTEID attributes that point to stanzas in transforms.conf.

In transforms.conf, look for stanzas referenced by props.conf that use REGEX or FILTER to evaluate the source field and then route events to a null queue (queue=nullQueue) or otherwise discard them. A regex or filter condition might be inadvertently matching sources shorter than 14 characters.

Compare the props.conf and transforms.conf files on the indexers handling data for Index 1 and Index 2 to identify the difference.

You can use the splunk btool command to inspect the effective configuration:

splunk btool props list <your_sourcetype> --debug
splunk btool transforms list --debug

This will show you which configuration files are contributing to the settings for your source type and transforms.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing