Universal Forwarder send logs to Spunk Cloud and AWS Cloudwatch

[Sec-Bolognese]

Hi - I need to be able to send copies of logs to both Splunk Cloud and an AWS Cloudwatch Log Group.  Is it possible to configure the Universal Forwarder to send logs from the same source to both locations?  If not, has anybody use UF and the Cloudwatch Agent to monitor the same log file - I'm worried about two products watching the same file.

[livehybrid]

Hi @Sec-Bolognese 
Ive achieved this before using the AWS Cloudwatch agent, as the others have mentioned this isnt really something you can do with the Splunk Universal Forwarder.

Step 1: Set Up IAM Permissions for Cloudwatch Agent if not already in place.

• Create (or use an existing) IAM role that has permissions for CloudWatch Logs.
• Ensure the role includes at least these actions:
  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents
  • logs:DescribeLogStreams
• If using EC2, attach the IAM role to your instance. Otherwise, provide credentials that have the above permissions.

Step 2: Install the CloudWatch Agent

For Amazon Linux, RHEL, or CentOS: sudo yum update -y sudo yum install -y amazon-cloudwatch-agent

For Ubuntu or Debian: sudo apt-get update sudo apt-get install -y amazon-cloudwatch-agent

(Alternatively, you can download the package directly from AWS if needed.)

---

Step 3: Create the CloudWatch Agent Configuration File

• Create a file at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json.
• Paste the following content into the file:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/opt/splunkforwarder/var/log/*",
            "log_group_name": "splunkforwarder-logs",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%m-%d-%Y %H:%M:%S.%f %z"
          }
        ]
      }
    }
  }
}

Note:

• Adjust "file_path" if you need a more specific file pattern (e.g., "/opt/splunkforwarder/var/log/*.log").

• "log_group_name" is the CloudWatch Logs group that will be used. If it doesn’t exist, the agent can create it (given sufficient permissions).

• "log_stream_name" uses {instance_id} as a placeholder. You can change this if desired.

• If your logs do not contain timestamps in the specified format, adjust or remove the "timestamp_format" setting.

• (Optional) You can also run the configuration wizard: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard Answer the prompts to generate a configuration file interactively.

---

Step 4: Start the CloudWatch Agent with Your Configuration

Run the following command to start the agent using your configuration file: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

In the above:

• -a fetch-config tells the agent to fetch the configuration.
• -m ec2 indicates that the agent is running on an EC2 instance (use -m onPremise if running elsewhere).
• -c file:... specifies the path to your configuration file.
• -s starts the agent.

---

Step 5: Verify That Logs Are Being Sent

• Check the CloudWatch Agent log file to ensure it started correctly: sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
• Log in to the AWS Console and navigate to CloudWatch - Logs.
• Look for the log group "splunkforwarder-logs" and verify that log streams and log events are appearing.

This should then allow you to send logs from /opt/splunkforwarder/var/log to CloudWatch Logs as well as your Splunk Cloud instance as required.

Please let me know how you get on and consider upvoting/karma this answer if it has helped.
Regards

Will

[PickleRick]

I don't know CloudWatch but from what I'm reading it uses either its own agent or you're pushing data to its API endpoint. Splunk's UF is obviously not a CloudWatch agent and it can only send out a simple syslog (or syslog-like) output. So your best bet would be probably using two separate agents. Watching the same file should not be that much of a problem (except for rare situations when monitoring a file with just one agent would be problematic).

[gcusello]

Hi @Sec-Bolognese ,

I don't know how AWS Cloudwatch runs, but, it's possible to dend logs from a Forwarder to Splunk Cloud and to a third party, following the instructions at 

https://docs.splunk.com/Documentation/Splunk/9.4.0/Forwarding/Routeandfilterdatad#Replicate_a_subset...

and

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd

Ciao.

Giuseppe