We're doing a POC to monitor a CloudWatch Log group's streams. It's been successfully indexing data. The only changes we've made from default settings are the index, source type, and regex. Whenever I make a change to the input config (via UI or aws_cloudwatch_logs_tasks.conf), the input stops collecting data. A sample change may be the polling frequency. Whether I restart Splunk, disable/re-enable the input, or re-create the input, it won't get anything. The only way that I've been able to solve the issue is to uninstall and reinstall the AWS TA.
To help, I've set the CloudWatch Logs input to DEBUG level logging. In Splunk_TA_aws_aws_cloudwatch_logs.log, I see that it's connecting to AWS and polling the correct group streams. The start and end times are correct for each stream's polling iteration. Even though there are messages in the stream, Splunk fails to realize this, and moves on to the next one. Modifying aws_cloudwatch_logs.py to show the number of results it got (line 245) confirms that it thinks there's nothing for it to do.
It's doubtful that it's an AWS/CloudWatch issue because reinstalling the TA solves the problem. The streams have events and the token I'm using has the proper IAM permissions.
I'm pretty puzzled by all of this and am open to suggestions as to how to fix it.
We're using Splunk Enterprise 6.2.5 with Splunk Add-on for Amazon Web Services 2.0.0.
