Missing events

[Wanpa]

We’ve confirmed that the Funding Auth Service successfully processed the transaction, the database status is ACQUIRED, and CloudWatch contains the expected logs. However, some of those log events are missing in Splunk. I’d like to understand where the log pipeline is breaking. It is anticipated that the root cause could be updating new splunk deployment server name with different port and updating splunk ip CIDRs on the terraform prod files.

[ITWhisperer]

Can you determine which logs are "missing"?

Are they from the same time period?

Are they the same format?

Are they the same event types?

etc.

You could do a bit more analysis to give us some clues as to where your issue(s) might be.

[Wanpa]

We are investigating an intermittent production log ingestion issue in our AWS ECS environment. The application processes transactions successfully—the order status is updated to ACQUIRED, the outbox event is generated and delivered, and no application exceptions are observed. The expected application log message, “Funding details marked as ACQUIRED and outbox event generated,” is present in AWS CloudWatch but is missing in Splunk for some FOIDs (for example, 00118def-0af3-a6b8-9188-c1725237e3fb). Other log events for the same transaction are searchable in Splunk, making the issue intermittent rather than a complete ingestion failure. We have also verified that similar transactions are successfully indexed in Splunk, while only specific FOIDs are affected. We recently updated the Splunk deployment server configuration and Splunk CIDR/IP configuration, although we have not yet established whether those changes are related to the issue. Based on this behavior, could you help identify where the most likely point of failure is in the CloudWatch-to-Splunk ingestion pipeline? Specifically, would you recommend focusing on forwarder/HEC connectivity, inputs.conf/outputs.conf, deployment server configuration propagation, host-specific collector issues, indexing/parsing, or another component? Any guidance on additional evidence or diagnostics that would help isolate the root cause would be greatly appreciated

[Wanpa]

Here’s a professional paragraph you can post on the Splunk Community or send to a Splunk engineer:

We are investigating an intermittent production log ingestion issue in our AWS ECS environment. The application processes transactions successfully—the order status is updated to ACQUIRED, the outbox event is generated and delivered, and no application exceptions are observed. The expected application log message, “Funding details marked as ACQUIRED and outbox event generated,” is present in AWS CloudWatch but is missing in Splunk for some FOIDs (for example, 00118def-0af3-a6b8-9188-c1725237e3fb). Other log events for the same transaction are searchable in Splunk, making the issue intermittent rather than a complete ingestion failure. We have also verified that similar transactions are successfully indexed in Splunk, while only specific FOIDs are affected. We recently updated the Splunk deployment server configuration and Splunk CIDR/IP configuration, although we have not yet established whether those changes are related to the issue. Based on this behavior, could you help identify where the most likely point of failure is in the CloudWatch-to-Splunk ingestion pipeline? Specifically, would you recommend focusing on forwarder/HEC connectivity, inputs.conf/outputs.conf, deployment server configuration propagation, host-specific collector issues, indexing/parsing, or another component? Any guidance on additional evidence or diagnostics that would help isolate the root cause would be greatly appreciated.

[PickleRick]

OK.

1. Do not just blindly copy-paste some AI slop. And please use some formating, break the text into paragraphs - generally, make the thing readable.

2. You posted in Observablility Cloud section of the forum but your AI-generated content is pointing to a Splunk Enterprise/Splunk Cloud specific features. So make up your mind - which is it?