topic Re: How to create a field at the forwarder layer for tracking... in Getting Data In

How to create a field at the forwarder layer for tracking...

a212830 — Tue, 31 Mar 2015 20:42:44 GMT

Hi,

I have a number of forwarders behind a load-balancer, and I want a way to see how the traffic is being distributed to each forwarder. Is there a way to create a field at the forwarder layer, and have it part of the event. Note that the host for the event is not the server name - the host is extracted from the message stream, so if a transforms is done, it can't over-ride the host that is being extracted.

btw - this really should be included in the product...going back and trying to determine where an event is coming from when you have a large environment can be a real pita....

Re: How to create a field at the forwarder layer for tracking...

dolivasoh — Tue, 31 Mar 2015 21:48:16 GMT

I suggest taking a look at creating default fields dynamically in http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Assignmetadatatoeventsdynamically . From here you may be able to create a new field called "forwarder" maybe? and assign the forwarder's hostname variable to the transforms for the field.

Re: How to create a field at the forwarder layer for tracking...

a212830 — Wed, 01 Apr 2015 01:32:57 GMT

I was able to create a field, but it's taking the value of the incoming message stream (tcp input). Is there a way to say use the hostname of the server that is running the forwarder? Any variable that might be available?

Re: How to create a field at the forwarder layer for tracking...

a212830 — Wed, 01 Apr 2015 13:45:20 GMT

Is this possible:

1) Add a monitor that runs a script that generates a lookup-file with the hostname of the server.
2) Use that lookup to populate the forwarder field (at the forwarder layer).

Re: How to create a field at the forwarder layer for tracking...

sloshburch — Wed, 01 Apr 2015 18:43:27 GMT

Making sure I have the architecture correct here, confirm if this is correct or not?

The flow is: devices -> tcp stream -> load balancer -> heavy forwarders -> splunk indexers.

The resulting event's 'host' field shows the load balancer. The events also have a field within them that shows the originating 'device'. The goal is to get the name of the splunk indexer that (or heavy forwarder?) that is processing the event.

Is that correct?

Re: How to create a field at the forwarder layer for tracking...

a212830 — Wed, 01 Apr 2015 18:54:26 GMT

Flow is correct. The reporting host is the load-balancer, but I extract the event host from the message stream. I want to use the hostname of the heavy forwarder for this new field.

Re: How to create a field at the forwarder layer for tracking...

sloshburch — Mon, 28 Sep 2020 19:23:53 GMT

I'm thinking you probably can do something with option 2. I haven't tested this myself but I would be inclined to see if this works:

Have a transforms.conf on the heavy forwarders that uses the "external_cmd" attribute to run a hostname, or the pyton equivalent, then deposit that as an indexed field. (This is technically a scripted lookup) Be sure to also set external_type accordingly.

You might have to then set the value using the DEST_KEY field.

BTW: I'm suspicious that if you are able to rename this question, something like "Identifying the heavy forwarder" or something more specific may get more of the splunkers to peek at it.

If you haven't already, be sure to submit a P4, feature request, to have this added as a default field.

Let us know how that approach works our for you so we can support you along the way.