HTTPOUT from an old Heavy Forwarder

[e_v_m]

Hi all,

I'd like to convert an existing, on prem, heavy forwarder to send it's data to a Cloud instance of Splunk using HTTPOUT (so it can be received using a HEC input). The catch is the heavy forwarder is running Splunk version 7.1.8 and it seems that is so old HTTPOUT is not something that can be added to the outputs.conf file (I can't find it in the documentation and it doesn't work when I try it).

This isn't intended as a permanent solution, just an interim until all the data arriving into the Heavy Forwarder is rerouted to the cloud directly. Sadly sending the data from the Heavy Forwarder to the cloud via S2S via the customer port number isn't allowed in this environment.

Any ideas?

Thanks

Eddie

[e_v_m]

Hi @gcusello @VatsalJagani @PickleRick 

Thanks for the response. I had a feeling that was going to be the answer. So I upgraded the HWF from 7.1 to 8.1.14, but then discovered whilst it could output data it ingested using HTTPOUT that doesn't include data fowarded to it from other universal forwarders (this is documented I discovered after). So in the end I made a sudden decision to change the HWF to a Universal Forwarder receiving data from other Universal Forwarders via old style S2S and send it to our cloud instance via HTTPOUT (and it's working)

This does leave me worrying my plan for migration to Splunk Cloud might have an issue: I planning to get rid of all on-prem servers, except for a newly build deployment server to managing the UF which could also act as a 'junior' heavy forwarder for anything unusual which needs to be collected by something on our internal network (I have a feeling DB connect functionality might be asked for soon, or possibly receiving a TCP stream, or even receiving data from old UF on old operating systems which can't handle new UF which support HTTPOUT). Will this plan not work when I need to send data to the Cloud from an Splunk Enterprise install using HTTPOUT?

Thanks all!

Eddie

[VatsalJagani]

@e_v_m - I think what you have in hand is not a quick question but instead a full migration planning and importantly complex one since many of components are of Splunk version which are no longer supported as @gcusello and @PickleRick has already pointed out. Also I think you have complex Splunk architecture involving many UFs and HFs with variety of Splunk versions and their roles at hand.

My suggestion would be to you is lay out full architecture diagram for yourself (not to share here), write down where is data flowing and how, splunk version everywhere, then what is possibility of upgrade at each component, etc information and then slowly work it through the migration plan.

[gcusello]

Hi @e_v_m ,

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[PickleRick]

That is a strange situation but not unheard of (sometimes an old piece of software is - for example - maintained as a part of a system with a very very high price tag and the system is very sensitive to any updates/reconfigurations - typical for OT systems, for example; or you might be interfacing with an ancient system for which you only have a very old app with modular input working only with old Splunk versions).

Yes, httpout (as a side note, it doesn't produce a HEC-compatible output; it produces s2s embedded in HTTP; to make things more confusing, both protocols are read by the same "HEC" port on the receiver's side and are controlled with a token configured the same way) appeared around 8.0 or even 8.2 so your 7.1 will not be able to use it.

The easiest solution would be - as already advised - to use another intermediate forwarder to "transform" your output stream to another format. You might try to use UF for that but HF->UF-> architecture is not recommended, not supported and might cause issues in certain scenarios (especially regarding useACK).

So, spin up a temporary HF and prepare to migrate away from the one you have.

[gcusello]

Hi @e_v_m ,

I am agree with  @VatsalJagani , at first I'd check if it's possible to upgrade the HF, mainly not because its old but because it's out of support so you cannot ask help to Splunk.

If you need a temporary solution, you could use an intemediate HF with an higher versione of Splunk, but in this way you don't solve the main issue: you have a very old system in your architecture and data continue to pass through it.

Ciao.

Giuseppe

[VatsalJagani]

@e_v_m - A simple answer would be to use Intermediate Splunk HF with a higher version of Splunk, which collects the data from this HF and forwards it over to Splunk Cloud. And you could use either Splunk TCP or Syslog as output.

But more importantly, you need to ask these questions to yourself and make sure if any of these are open for implementation:

• Why can't Splunk HF be upgraded?
• Is there a reason to keep this data on a different port?

I hope this helps!!