topic Re: How do I Host name override HTTP Event Collector? in Getting Data In

How do I Host name override HTTP Event Collector?

dsfyxcasdcertzu — Wed, 05 Apr 2023 16:58:25 GMT

Hello,

We're running localhost Http Event Collectors on UF for Docker Containers on the same host. However I'm unable to override the hostname from these Events. Unfortunately there is no flag to do so in the docker daemon. Therefor I've tried to do it on the Forwarder as well as on the indexer. Both unsuccessful.

On the Forwarder:

/opt/splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf

[http] host = wantedHostName disabled=0 port = 8088 enableSSL=0 dedicatedIoThreads=2 maxThreads = 0 maxSockets = 0 useDeploymentServer=0

/opt/splunkforwarder/etc/apps/splunk_httpinput/local/inputs.conf

[http://localhost] host = wantedHostName description = <desc> disabled = 0 index = main token = <token> useACK = false

On the indexer:

/opt/splunk/etc/system/local/props.conf

[host::badHostName] TRANSFORMS-badhost = badHostName

/opt/splunk/etc/system/local/transforms.conf

[badHostName] DEST_KEY = MetaData:Host REGEX = * FORMAT = host::wantedHostName

None of these work. Can someone please help us out?

Cheers!

Re: Host name override HTTP Event Collector

scelikok — Wed, 05 Apr 2023 11:11:08 GMT

Hi @dsfyxcasdcertzu,

To run transform for HEC inputs, your client application must use "raw" endpoint on HEC output. If you are using "event" endpoint it is not possible to change anything on the data.

Re: How do I Host name override HTTP Event Collector?

dsfyxcasdcertzu — Wed, 05 Apr 2023 17:47:31 GMT

Thank you @scelikok for your quick response!

So essentially there is no way to override the host name when using the default docker/splunk logging driver.
I've tried the extended docker-logging-plugin with the (undocumented) path flag in combination with the raw endpoint of HEC but this would prerequisite to transform the complete transactions content.

Re: How do I Host name override HTTP Event Collector?

scelikok — Thu, 06 Apr 2023 04:08:45 GMT

Hi @dsfyxcasdcertzu,

Yes, raw endpoint causes message to change. You can try using env field for setting correct hostname. It will not change host value but you will be able to query the real hostname.

https://docs.docker.com/config/containers/logging/splunk/

Re: Host name override HTTP Event Collector

PickleRick — Thu, 06 Apr 2023 04:17:27 GMT

Sorry, but I strongly disagree here. HEC is just another input and it the event goes through most of the processing stages. It bypasses line breaking and - unless explicitly enabled by url parameter - date parsing.

Re: How do I Host name override HTTP Event Collector?

PickleRick — Thu, 06 Apr 2023 04:23:04 GMT

What kind of forwarder do you have there? UF doesn't have HEC inputs. HF parses events so they're sent as parsed to the downstream indexers and are _not_ parsed anymore.

Re: How do I Host name override HTTP Event Collector?

dsfyxcasdcertzu — Thu, 06 Apr 2023 08:03:09 GMT

Hi, @PickleRick ,

I'm indeed running UF v9.0.4. on a Linux client. I know that HEC on UF is not supported on paper but since the app was pre-shipped in the bundle I've tried it and it works just fine. Also in on other (Linux-) machines.

This way we don't have to mess with certificates for docker logs because the port is http and localhost only on the client with it the outputs getting securely forwarded with the other outputs of the UF.

Nonetheless, transforming attempts were done on the indexer.

Best

Re: How do I Host name override HTTP Event Collector?

dsfyxcasdcertzu — Thu, 06 Apr 2023 08:18:05 GMT

@PickleRick wrote:
HF parses events so they're sent as parsed to the downstream indexers and are _not_ parsed anymore.

This would be the answer btw. I'm confident that UF does it in the same way when using the `\event` endpoint. However Transforms on UF are not supported as far as I know.

Re: How do I Host name override HTTP Event Collector?

PickleRick — Thu, 06 Apr 2023 09:10:18 GMT

I must say I'm puzzled. HEC is not supposed to work on UF. If it is, I'm wondering myself what data it sends downstream because since it's a non-parsing component it should just send cooked data and parsing should be done on the first "heavy" component in event's path. So if you're sending to indexers, the events should be properly modified there according to props/transforms.

Anyway, the default metadata fields manipulation can sometimes be tricky. I always add WRITE_META to the transform stanza.

Re: Host name override HTTP Event Collector

scelikok — Thu, 06 Apr 2023 10:56:53 GMT

Hi @PickleRick,

It is not stated in documents but HEC event endpoint skips TRANSFORMS.

Re: Host name override HTTP Event Collector

PickleRick — Thu, 06 Apr 2023 12:22:12 GMT

Sorry, but no. Whole functionality of SC4S (or my rsyslog-based solution) depends on transforms working properly on HEC-based events. So no. Transforms do work normally on HEC-ingested data.