HF licensing when License Manager is not reachable due to network restrictions

[ibrahim1]

We have a distributed on-prem Splunk environment with strict network segmentation between sites.

Scenario:

Site B:

• Source machines run Universal Forwarders
• Logs are sent to a Heavy Forwarder that also acts as a Deployment Server

Site A:

• Indexer Cluster and all management components, including the License Manager (let say zone E)
• A separate Heavy Forwarder that receives data from Site B (let say zone F)

Data flow:

UF (Site B) → HF (also act as DS) (Site B) → HF (Site A) → Indexers

Constraint:

Due to network policy, Site B HF can communicate only with Site A HF.
Direct connectivity from Site B to Site A management components (License Manager) is not allowed.

Problem:
The Site-B Heavy Forwarder (Splunk Enterprise) requires access to the License Manager, but this connectivity is blocked by design.

Question:

• Is there a supported approach to handle HF licensing when direct access to the License Manager is not possible?
• Has anyone encountered a similar setup and how was it handled?
• Any guidance or real-world experience would be appreciated.

[PickleRick]

OK. You've already gotten some hints here.

Generally, a HF needs a license. You could use a forwarding-only license but then you'd use the DS functionality.

So you could call out to your Splunk sales contact for a 0-bytes license. It's normally meant for Cloud deployments for exactly those scenarios - you need to have a DS and/or HF on premises whereas the "main" part of your Splunk infrastructure is the Splunk Cloud service. True, your case is a bit different but boils down to the same thing - the need to be able to use a limited subset of functionalities in a site without actually doing any indexing/searching work in that site.

You might be able to get a "blessing" for using the same production license in both sites as long as there is absolutely no indexing going on in the B site. But for that you need to talk with your local Splunk sales team.

Generally - HF/DS functionalities need a license and somehow you must provide that host with one. And generally it's best to talk with your local sales team how best to tackle the issue.

I'm still perplexed however why would normal event stream connectivity be allowed but LM connectivity - not.

[asimit]

Hey @ibrahim1 , tough spot with that network seg - seen it before! Your Site B HF def needs Enterprise license cuz its parsing/processing data. UFs are free but HFs gotta phone home to LM. Here's what actually works:

Option 1: UF Intermediate Forwarder (cleanest - my rec)
Generally most of the time we follow "ntermediate Forwarder" setup to route logs form multiple network sites,  Replace Site B HF with Universal Forwarder. Always free, no LM contact EVER needed.

## Example conf for New Site B UF, change as per your details
inputs.conf:
[splunktcp://9997]
disabled = 0

outputs.conf:
[tcpout]
defaultGroup = sitea_hf

[tcpout:sitea_hf]
server = siteA_HF_IP:9997

Source UFs point to new Site B UF:9997. Site A HF does all parsing. Data flows: UF (Site B)→UF(Site B)→HF(Site A)→indexers(Site A). Worked perfect in my DMZ setup.

Lose Deployment Server role tho - move that elsewhere or use UF phonehome.

Option 2: Selective 8089 firewall rule (if u must keep HF)
Ask netsec for just SiteB_HF_IP → SiteA_LM_IP:8089. License heartbeat only (every 30min). Security teams usually approve - low risk vs full mgmt ports.

# server.conf at Site B HF
[license]
master_uri = https://siteA_LM:8089

Option 3: License pooling (if 8089 opens up)
Create "forwarders" pool on LM, add Site B HF as slave. Still needs 8089 tho.

What I'd do: Go UF intermediate. Zero license drama, lighter resource use, same data flow. Test on one source first 

Please give karma 👍 for support 😁 happly splunking .... 😎

[ibrahim1]

Hi @asimit,

Thank you for taking time answering my questions. Your rec solution makes sense, the option 1. But as you point it out, I'll lose the DS role.

So, If I go with UF intermediate on Site B, how can I manage the machines configurations centrally without DS?

Can I use Site A HF also as a DS and manage configurations from there, will it work?

[asimit]

Hi @ibrahim1 ,

Site A HF as DS? Short answer: NO - network blocks prevent Site B UF phonehoming to it (8089 outbound needed). UFs poll DS every 30min-ish, same port issue as LM.​

For single UF intermediate: Honestly dont need DS much. Configs super simple:

Manual SCP/ansible/rsync changes once a month? Same effort w/ or w/o DS for one box. No biggie.

If multiple Site B UFs (or source UFs too):

• Phonehome workaround: Site B UF → Site A HF:8089 blocked? Use Deployment Server Cluster but needs shared storage between DSs + load balancer. Overkill for forwarders.​

• Forwarder mgmt proxy hack (what I've done): Site A HF relays configs? Nah Splunk no built-in for that. But script it:

  1. Ansible/Puppet from central → Site A HF

  2. Site A HF cron job bundles apps → sends via tcpout to Site B UF (as cooked data)

  3. Site B UF inputs.conf catches it, extracts/unpacks to etc/apps/

Kinda janky but works airgapped.

Best simple way: Keep source UFs phonehoming to Site A DS/HF (if u open 8089 Site B sources -> Site A). Intermediate UF stays dumb/manual.

Or stick w/ HF + 8089 whitelist (one port, easy approval).

Single intermediate? Skip DS drama, manual configs fine.
Multiple? Ansible > DS in seg nets IMO.

[gcusello]

Hi @ibrahim1 ,

there is a Forwarder License that you can enable on your HFs instead of the connection to the License Master.

But there's a restriction: if on the HF you must use DB-Connect or and Add-Ons that use APIs (e.g. Azure or AWS) or use it as a Deployment Server, you cannot use this kind of license.

So, the solution could be one of the following:

• open the connection between the HF/DS/B and the License Master,
• Create a new DS on Site B and open the connection (both 8089 and 9997 ports) with the LM/A only for this server,
• Use the DS on the Site A to manage UFs and HFs on Site B, but in this case you should open more connections!
• use the HF/A as License Master: but you can use this solution only if the License Master has less than 50 clients, but anyway, you have to open the connection between the HF/LM/A and the HF/DS/B also on the 8089 Port, that's the required port for management and that usually closed, in addition to the 9997 data port,
• add a minimal License (also 500 MB/day) on the HF/DS/B.

Ciao.

Giuseppe

[ibrahim1]

Hi @gcusello,

Thank you for taking time answering my questions. Can you provide a little detail on your point 4;  how can I use site-a HF as LM?

[gcusello]

Hi @ibrahim1 ,

you should configure one of HF in Site A as License Master for all the Indexers and other Splunk Server in SiteA.

Then open the firewall routes on port 8089 between HF-A and HF-B, in this way you can use this HF as Liense Master also for the HF-B.

Ciao.

Giuseppe

[livehybrid]

Hi @ibrahim1 

There are a couple of ways you could combat this, both of which I have seen at other customers.

1) Request a 0-byte license from Splunk Support - Explain your situation that you cannot connect from Site B to Site A and that you have Splunk components in Site B which require a license. Because Site B isn't actually indexing anything, it's only forwarding the data. You don't actually need a licence with a particular volume limit, therefore a 0-byte licence will be sufficient. 

2) you could install the same licence on site A and B, however you may want to double-check with your Splunk account team. I have seen this done before, and Splunk did not have an issue with this, so long as no data was being ingested in Site B. I think the only time that you would get a warning within Splunk is if you had multiple index clusters pointing to different licence servers with the same licence key. But for HF, you should not have this problem. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo]

I don't believe that you can install same license on both sides. Basically it's deny in license rules and there could be issue with collecting internal logs which contains that information. In that case you have 72h to fix issue or splunk disable your searches at least.

But you can ask that Splunk Support split to your licenses. Or ask DS/HF license from them.

Currently the easiest solution is behind this link https://splunk.my.site.com/customer/s/article/0-byte-license-for-Deployment-Server-or-Heavy-Forwarde... You could just download that 0-byte license there.

[ibrahim1]

Hi @livehybrid,

Thank you for taking time answering my questions. I'll definitely check with Splunk Support and explore both options.

However, can you confirm if there is any cost associated with 0-byte license?

[erikwie]

Hi @ibrahim1 

I have set up a few Splunk installations that fits your design
UF (Site B) → HF (also act as DS) (Site B) → HF (Site A) → Indexers

I have them set up with the original license file on the site B HF, as far is it knows there isn't any License Master in the system and it would not have any issue with the license already been used somewhere else.

I have also done this for Splunk Cloud customers using the 0-byte license, typically sending logs from another zone like DMZ inwards to a onsite Intermediate forwarder before sending it to the cloud.

So both ways will work.

In my experience there is no cost to the 0-byte license, but you need to have an active license in order to send a ticket to Splunk support.