SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

[SplunkDash]

Hello,

Do we have any SPLUNK TA that can write logs from SPLUNK Server with HF to AWS S3/SQS.  Any recommendation will be highly appreciated, thank you! 

[PickleRick]

If you look into outputs.conf specs, you'll see that it supports both SQS output as well as RFS output which should be able to write into S3 buckets. Never used them myself though so I have no idea how they work and whether they require HF or if they will work with UF as well (I suspect the former).

[richgalloway]

Does it have to use an HF?  The Export Everything app (https://splunkbase.splunk.com/app/5738) can write to S3

[SplunkDash]

@richgalloway 

Thank you so much for your quick response.

It's not exporting SPLUNK search results, it about writing Logs into S3 bucket using SPLUNK TA. For Example, we have some Application logs within server, we would prefer to use SPLUNK TA to write those logs into S3 Buckets from there and ingest data from S3/SQS. This server has the HF install on them. We cannot perform direct ingestion from that server due to security reason.  Any thoughts or recommendations

[richgalloway]

Please tell us more about the environment.  Can the server relay data to Splunk via an intermediate forwarder?  Why is an HF installed instead of a Universal Forwarder (UF)?  UFs have a much smaller footprint and attack surface.

[SplunkDash]

@richgalloway 

I think HF/UF doesn't have any role here; main use case: we have a server need to write data from that server to AWS S3 Bucket; do we have any TA?

[richgalloway]

Here's an untested idea.  Install an HF on the server and use Splunk's Ingest Actions feature to write the data to S3.  It's not clear if the HF will be happy only writing to S3 or if it also will want to send to an indexer.

See https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/DataIngest#Heavy_forwarders_managed_through_... for details, including the need for a Deployment Server.