Custom object store for on-prem edge?

[garethhumphrgkc]

Hi,

I was excited to see edge processor has been made available in Splunk enterprise on-prem - this will be super handy for one of our installations where cloud isn't an option.

In spiking a PoC however, I noticed that the S3 destinations are _only_ able to speak to AWS - there is no way to use a custom endpoint URL, so no way to send to our own S3-compatible object store.  I had a good look at the deployed config and API calls, and there is no endpoint configured anywhere there, so it appears to be baked into the code.  Am I right about that?

We're using Splunk enterprise because cloud services aren't possible for this deployment - naturally that includes AWS as well as Splunk cloud, so suddenly edge became a whole lot less useful for us.

Does anyone know if this limitation is by design, and will it be changed in the near future?

I suspect this is actually a pretty big limitation for a lot of on-prem edge users and quite a surprising omission.  I'm keen to find out what the deal is!

[PickleRick]

Well, the destination is literally called "Amazon S3" and has AWS-specific parameters like region. So it's kinda expected. I'm not saying that it wouldn't be nice to be able to send to other destinations. It's just that with a name like this I wouldn't expect the ability to push to on-prem.

[Omar4]

You're not wrong—I ran into the same wall when testing this on-prem. The S3 destination really does appear to be hardcoded to AWS endpoints, no custom endpoint URL option anywhere in the UI or config.

It's a strange omission given that the whole point of on-prem Edge is for environments that can't or won't use cloud services. If you're running Splunk Enterprise because AWS isn't an option, then a destination that only speaks to AWS is essentially useless.

I'd guess it's a case of the feature being lifted from the cloud version without much adaptation for on-prem realities, rather than a deliberate design choice. But I can't say for certain.

Might be worth raising it directly with your Splunk account team or in an Idea on the community—if enough on-prem users hit the same wall, it's the kind of thing they may prioritise. In the meantime, I haven't found a workaround that doesn't involve some kind of intermediary (a local service that receives the data and forwards it to the S3-compatible store). Not ideal, but it's an option if you're determined to make it work.

Curious to hear if anyone else has cracked it.

[garethhumphrgkc]

Yeah, my thoughts exactly - a very strange omission, and your reasoning behind it is sound.  You can't even use other cloud providers like Azure or GCP, so surely an oversight.

My current plan is to just deploy Cribl as an interpreter - HEC in, S3 out - but that adds an extra moving part for no good reason.

I've raised it with partner support, but that hasn't been great since the purchase so I'm not optimistic of a good outcome.  I'll let you know if I hear anything back.