All Apps and Add-ons

Evident.io App for Splunk: Can Splunk and the Lambda functions be on different subnets?

Explorer

We are trying to integrate Evident.io logging with Splunk using Evidentio->SNS->Lambda->Splunk. Following the steps are listed in http://docs.evident.io/#organization , basically it talks about

1)Installing the Evident.io App for Splunk
2)Creating a HTTP Event Collector token in Splunk
3)Creating Lambda function for splunk integration
4)Evident.io SNS integration
5)Lambda and SNS integration

In step 3, during creation of lambda function, procedure is to encrypt the “HTTP Event Collector token” generated in Splunk, using KMS service and to use the encrypted token in the lambda function. When the Lambda function runs, it uses the KMS service to decrypt the token and to talk to Splunk servers.

With our current network design, Splunk instances are in a private subnet with internet connection through proxy from our corporate network. When the lambda function is created in default VPC (No VPC (if NO VPC is selected all AWS Lambda functions run securely inside a default system-managed VPC.)) function is able to talk to KMS service and unable to talk to Splunk servers. When function is created in subnet where the Splunk instance are, function is unable to talk to KMS and failing.

In this scenario, Lambda function needs access to internet to use KMS and access to Splunk servers in private subnet.

Can the Splunk-logging Lambda function take the proxy setting to make the API calls to AWS KMS for decryption of the “HTTP Event collector token”?

Can there be situations where Splunk and the Lambda function are on different subnets?

0 Karma
1 Solution

Explorer

We were able to create the Lambda function without using KMS to encrypt the token for the HEC. While less than ideal it has us up and collecting Dome9 events in the Splunk Cluster running in or AWS VPC.

View solution in original post

Splunk Employee
Splunk Employee

@Gerrykahn:
To allow a VPC-enabled Lambda function access to public internet, you need to attach it to a private subnet with internet access though a NAT instance or a VPC NAT Gateway. Create this new private subnet in the same VPC where you Splunk instances live to give Lambda function connectivity to both:

  • KMS service: via public internet (through NAT instance/gateway)
  • Splunk servers: via local route

A side benefit of a dedicated private subnet is that you'll have a larger pool of private IP addresses available to Lambda to scale as it sets up network interfaces for your Lambda functions.

With respect to encrypting the Splunk HEC token: with Lambda support of environment variables (released just days after this question was posted), Lambda automatically encrypts these variables by default (using KMS), and decrypt them for you per function invocation. That may be sufficient depending on your security requirements. You can also manually encrypt the environments variables before you deploy Lambda, as you have previously done, where Lambda function code will have to decrypt them at run-time and will therefore need access to KMS.

By the way, Splunk has released several basic Lambda blueprints to collect data from different AWS services such as DynamoDB, Kinesis and CloudWatch Logs including a generic logging one. There's also a step-by-step walkthrough on how to set them up. Thought you may be interested in those as well.

0 Karma

Explorer

We were able to create the Lambda function without using KMS to encrypt the token for the HEC. While less than ideal it has us up and collecting Dome9 events in the Splunk Cluster running in or AWS VPC.

View solution in original post