topic Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS in Getting Data In

SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

SplunkDash — Thu, 18 Jan 2024 17:38:55 GMT

Hello,

Do we have any SPLUNK TA that can write logs from SPLUNK Server with HF to AWS S3/SQS. Any recommendation will be highly appreciated, thank you!

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

richgalloway — Thu, 18 Jan 2024 18:06:03 GMT

Does it have to use an HF? The Export Everything app (https://splunkbase.splunk.com/app/5738) can write to S3

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

SplunkDash — Thu, 18 Jan 2024 18:21:42 GMT

@richgalloway

Thank you so much for your quick response.

It's not exporting SPLUNK search results, it about writing Logs into S3 bucket using SPLUNK TA. For Example, we have some Application logs within server, we would prefer to use SPLUNK TA to write those logs into S3 Buckets from there and ingest data from S3/SQS. This server has the HF install on them. We cannot perform direct ingestion from that server due to security reason. Any thoughts or recommendations

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

richgalloway — Thu, 18 Jan 2024 19:42:29 GMT

Please tell us more about the environment. Can the server relay data to Splunk via an intermediate forwarder? Why is an HF installed instead of a Universal Forwarder (UF)? UFs have a much smaller footprint and attack surface.

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

SplunkDash — Thu, 18 Jan 2024 19:56:10 GMT

@richgalloway

I think HF/UF doesn't have any role here; main use case: we have a server need to write data from that server to AWS S3 Bucket; do we have any TA?

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

richgalloway — Sat, 20 Jan 2024 18:32:06 GMT

Here's an untested idea. Install an HF on the server and use Splunk's Ingest Actions feature to write the data to S3. It's not clear if the HF will be happy only writing to S3 or if it also will want to send to an indexer.

See https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/DataIngest#Heavy_forwarders_managed_through_a_deployment_server for details, including the need for a Deployment Server.

Re: SPLUNK TA to Write Log from SPLUNK HF server to S3 and SQS

PickleRick — Sat, 20 Jan 2024 22:31:35 GMT

If you look into outputs.conf specs, you'll see that it supports both SQS output as well as RFS output which should be able to write into S3 buckets. Never used them myself though so I have no idea how they work and whether they require HF or if they will work with UF as well (I suspect the former).