Getting Data In

SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

SplunkDash
Motivator

Hello,

Do we have any SPLUNK TA that can write logs from SPLUNK Server with HF to AWS S3/SQS.  Any recommendation will be highly appreciated, thank you! 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you look into outputs.conf specs, you'll see that it supports both SQS output as well as RFS output which should be able to write into S3 buckets. Never used them myself though so I have no idea how they work and whether they require HF or if they will work with UF as well (I suspect the former).

richgalloway
SplunkTrust
SplunkTrust

Does it have to use an HF?  The Export Everything app (https://splunkbase.splunk.com/app/5738) can write to S3

---
If this reply helps you, Karma would be appreciated.

SplunkDash
Motivator

@richgalloway 

Thank you so much for your quick response.

It's not exporting SPLUNK search results, it about writing Logs into S3 bucket using SPLUNK TA. For Example, we have some Application logs within server, we would prefer to use SPLUNK TA to write those logs into S3 Buckets from there and ingest data from S3/SQS. This server has the HF install on them. We cannot perform direct ingestion from that server due to security reason.  Any thoughts or recommendations

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please tell us more about the environment.  Can the server relay data to Splunk via an intermediate forwarder?  Why is an HF installed instead of a Universal Forwarder (UF)?  UFs have a much smaller footprint and attack surface.

---
If this reply helps you, Karma would be appreciated.

SplunkDash
Motivator

@richgalloway 

I think HF/UF doesn't have any role here; main use case: we have a server need to write data from that server to AWS S3 Bucket; do we have any TA?

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Here's an untested idea.  Install an HF on the server and use Splunk's Ingest Actions feature to write the data to S3.  It's not clear if the HF will be happy only writing to S3 or if it also will want to send to an indexer.

See https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/DataIngest#Heavy_forwarders_managed_through_... for details, including the need for a Deployment Server.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...