Splunk Search

Where does HEC push come in from?

ddrillic
Ultra Champion

A customer is asking:

"How can we tell where an HEC push is actually coming in from? or is that just not logged anywhere?"

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi ddrillic,

I started to use a props.conf, transforms.conf setting to work around exactly this issue. It will add a meta data field containing the host that parsed the event therefore you will always know which Splunk instance parsed the events or where it came from.

Here is my config:
props.conf

[default]
TRANSFORMS-000-add-relay-info-to-meta = add-relay-info-to-meta

transforms.conf

[add-relay-info-to-meta]
FORMAT = splunk_hwf::<hostNameHere>
REGEX = .
WRITE_META = true

You can then search for index=_internal splunk_hwf::* to see the Splunk instance that pared the events.
The down side is that the hostname value needs to be hard coded, but I have an app that works around this as well 😉

Also be aware this will only work on non- INDEXED_EXTRACTIONS events, if you use a default parsing pipeline setup.

Hope this helps ...

cheers, MuS

UPDATE

The correct answer would be changing the connection_host in inputs.conf for the according [http...] stanza

connection_host = [ip|dns|proxied_ip|none]
* Specifies the host if an event doesn't have a host set.
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system
  sending the data.
* "proxied_ip" checks whether an X-Forwarded-For header was sent
  (presumably by a proxy server) and if so, sets the host to that value.
  Otherwise, the IP address of the system sending the data is used.
* "none" leaves the host as specified in the HTTP header.
* No default.

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi ddrillic,

I started to use a props.conf, transforms.conf setting to work around exactly this issue. It will add a meta data field containing the host that parsed the event therefore you will always know which Splunk instance parsed the events or where it came from.

Here is my config:
props.conf

[default]
TRANSFORMS-000-add-relay-info-to-meta = add-relay-info-to-meta

transforms.conf

[add-relay-info-to-meta]
FORMAT = splunk_hwf::<hostNameHere>
REGEX = .
WRITE_META = true

You can then search for index=_internal splunk_hwf::* to see the Splunk instance that pared the events.
The down side is that the hostname value needs to be hard coded, but I have an app that works around this as well 😉

Also be aware this will only work on non- INDEXED_EXTRACTIONS events, if you use a default parsing pipeline setup.

Hope this helps ...

cheers, MuS

UPDATE

The correct answer would be changing the connection_host in inputs.conf for the according [http...] stanza

connection_host = [ip|dns|proxied_ip|none]
* Specifies the host if an event doesn't have a host set.
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system
  sending the data.
* "proxied_ip" checks whether an X-Forwarded-For header was sent
  (presumably by a proxy server) and if so, sets the host to that value.
  Otherwise, the IP address of the system sending the data is used.
* "none" leaves the host as specified in the HTTP header.
* No default.

MuS
SplunkTrust
SplunkTrust

Update ping - I misunderstood the question, but this will give you a two in one answer/solution 🙂

0 Karma

ddrillic
Ultra Champion

Very interesting information as always @MuS.

0 Karma

ddrillic
Ultra Champion

@Mus, my buddy says -

-- If the host isn't specified for data that is coming through HEC, then it takes the VIP/HF hostname…which is what we don't want.

Is it right?

0 Karma

MuS
SplunkTrust
SplunkTrust

Just the messenger here 😉 The docs say:

* Specifies the host if an event doesn't have a host set.
 * "ip" sets the host to the IP address of the system sending the data.
 * "dns" sets the host to the reverse DNS entry for IP address of the system
   sending the data.

It is most likely true, because the default for connection_host is empty and therefore you would get the hostname of the instance running the HEC input.
I never used this setting nor had to play with it, because we use dedicated HEC inputs and also only have one sender for each HEC input.

cheers, MuS

0 Karma

ddrillic
Ultra Champion

@MuS - we don't set the connection_host parameter and the host field ends up to be one of the indexers, but let me check....

0 Karma

Eldenhanjoel
Explorer

Did you create multiple tokens and want to find out which token is sending in the logs or do you want to know from which host you are receiving logs from?

0 Karma

Eldenhanjoel
Explorer

| tstats count where index=* sourcetype=X by host
| sort 0 -count

0 Karma

ddrillic
Ultra Champion

The second - from which host you are receiving logs from?

0 Karma

VatsalJagani
Champion

@ddrillic

I think you want to understand the working of HEC. The easiest way to understand HEC is that consider it as rest-endpoint.
Just like rest-endpoint, it is always in listening mode as and when someone tries to access it will respond. In case of HEC when someone sends an event, HEC receives as a request and get the parameters from it and stores those as an event in Splunk.

I hope you understand what I'm trying to say.

0 Karma

ddrillic
Ultra Champion

Fair enough, the question is whether any data about the sender is stored somewhere...

0 Karma

VatsalJagani
Champion

Not sure about sendor as if two sendor is using same token then they are considered to be same in Splunk. But for debugging purpose you can use index="_introspection" sourcetype=http_event_collector_metrics.

ddrillic
Ultra Champion

Great. index="_internal" <HEC Key> was also useful ; -)

VatsalJagani
Champion

Hey right, thanks for sharing this information.

0 Karma
Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...