What is the best practice for configuring a Splunk Forwarder 5.0.3 for custom fields to be appended to the source?

[JKnightSplunk]

Hi all,

I'm looking to add some custom fields to the Splunk Forwarder, but am struggling to find the a way of achieving this and determine the best way for performance.

Could I please get an example of configuring this through the forwarder for two additional fields to be appended to the source only to be retrieved through search with these values. Additionally, these two fields should be able to be populated through a script as this value will reside within a file on the server the forwarder resides.

Thanks in advance for any input.

[lguinn2 · Accepted solution]

Splunk has a server name that it stores in $SPLUNK_HOME/etc/system/local/server.conf and in $SPLUNK_HOME/etc/system/local/inputs.conf

The server name in inputs.conf is the one that is automatically set as the host field for data, unless you specifically override it:

[default]
host = the_host_name

In AWS, I believe that you can have a "run once" script that runs when a new instance is started for the first time. That would be the perfect place to set the host name to whatever you want. And the host name will be part of the data that is collected for every input.

On the search head (or indexer if there is no search head), you can create search time fields based on the host name using props.conf.

Actual "custom fields" (for example, where additional fields are added to a log file) are not easy to do in Splunk. Another approach, which might be better: collect instance information as a separate input, and then use it as part of a search or a lookup table. The best way to do this is probably a scripted input. You write a script that collects the data that you want; the input then runs the script at intervals and Splunk indexes the output of the script.

[lguinn2]

Can you explain the data a little more specifically? What is in the source, and what are the additional fields? Can the additional fields be computed from any information in the source input?

[JKnightSplunk]

Hi lguinn,

The device is instantiated through an AWS AMI which has the Splunk forwarder installed. I've configured the Splunk host name to give more details than the AMI number such as - which will only need to be set once along with some additional data such as the IP and instance-id. We'd prefer this additional data was able to be queried via fields of these names.

Any best way of configuring the host name without external tools would be appreciated too.