Getting Data In

How to create a field at the forwarder layer for tracking...

a212830
Champion

Hi,

I have a number of forwarders behind a load-balancer, and I want a way to see how the traffic is being distributed to each forwarder. Is there a way to create a field at the forwarder layer, and have it part of the event. Note that the host for the event is not the server name - the host is extracted from the message stream, so if a transforms is done, it can't over-ride the host that is being extracted.

btw - this really should be included in the product...going back and trying to determine where an event is coming from when you have a large environment can be a real pita....

sloshburch
Splunk Employee
Splunk Employee

I'm thinking you probably can do something with option 2. I haven't tested this myself but I would be inclined to see if this works:

Have a transforms.conf on the heavy forwarders that uses the "external_cmd" attribute to run a hostname, or the pyton equivalent, then deposit that as an indexed field. (This is technically a scripted lookup) Be sure to also set external_type accordingly.

You might have to then set the value using the DEST_KEY field.

BTW: I'm suspicious that if you are able to rename this question, something like "Identifying the heavy forwarder" or something more specific may get more of the splunkers to peek at it.

If you haven't already, be sure to submit a P4, feature request, to have this added as a default field.

Let us know how that approach works our for you so we can support you along the way.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Making sure I have the architecture correct here, confirm if this is correct or not?

The flow is: devices -> tcp stream -> load balancer -> heavy forwarders -> splunk indexers.

The resulting event's 'host' field shows the load balancer. The events also have a field within them that shows the originating 'device'. The goal is to get the name of the splunk indexer that (or heavy forwarder?) that is processing the event.

Is that correct?

0 Karma

a212830
Champion

Flow is correct. The reporting host is the load-balancer, but I extract the event host from the message stream. I want to use the hostname of the heavy forwarder for this new field.

0 Karma

dolivasoh
Contributor

I suggest taking a look at creating default fields dynamically in http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Assignmetadatatoeventsdynamically . From here you may be able to create a new field called "forwarder" maybe? and assign the forwarder's hostname variable to the transforms for the field.

0 Karma

a212830
Champion

I was able to create a field, but it's taking the value of the incoming message stream (tcp input). Is there a way to say use the hostname of the server that is running the forwarder? Any variable that might be available?

0 Karma

a212830
Champion

Is this possible:

1) Add a monitor that runs a script that generates a lookup-file with the hostname of the server.
2) Use that lookup to populate the forwarder field (at the forwarder layer).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...