Best architecture for collecting logs from a third-party app hosted on AWS via API calls (Splunk Enterprise On-Prem)

[Nicolas2203]

Hello Splunk community,

I’m working with Splunk Enterprise On-Prem and have three Heavy Forwarders (HFs) in my environment. I need to ingest logs from an application hosted on Amazon AWS, but here’s the challenge:

• The application is developed by a third party, and I have no access to their AWS subscription.
• The logs can be retrieved either by manual export from the application directly or via API calls.

The most efficient approach seems to be automated collection via API calls.

My questions:

1. What is the best architecture for this scenario?

   • Should I host a Python script on a Heavy Forwarder in a DMZ to collect logs via API and then forward them to another HF outside the DMZ, which will send them to the indexers?
   • Is this considered a Splunk best practice?

2. Are there better alternatives for this use case? 

I’ve never had to deal with this type of architecture before, so any guidance, best practices, or examples would be greatly appreciated.

Thanks in advance for your help!

[livehybrid · Accepted solution]

Hi @Nicolas2203 

It seems that building a custom application with a Python module input that can collect the data from the API would be the most appropriate approach here. 

Check out https://splunk.github.io/addonfactory-ucc-generator/ which is a tool to help build custom apps like this, there is also a sample app which might give some insight on how it works, this is taken from a Conf talk I did on creating a simple API app in 2023. https://github.com/livehybrid/conf23-dev1091b/

This blog post is might also be useful, https://www.splunk.com/en_us/blog/customers/managing-splunk-add-ons-with-ucc-framework.html

Okay. In terms of where to actually deploy the app, ultimately this depends on your architecture. But you will need somewhere that can access the API of the application and then forward the data to your Splunk environment.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Nicolas2203]

Hi @livehybrid  Thank you so much for your answer and for sharing those resources  I really appreciate it.

Regarding where to put the app, I believe hosting it on a HF will be the best approach, and I'll make sure that the HF have an access on the AWS hosted app.

Thanks again for your help

Nicolas