topic Re: Host override issues in Getting Data In

Host override issues

wwwdrich — Mon, 28 Sep 2020 12:54:17 GMT

I am trying to override the host field based on an element in the source path. This is data that is being forwarded from a universal forwarder and we want the host the data originally came from as the host field, not the hostname of the forwarder.

To date, I have tried modifying etc/system/local/transforms.conf and props.conf both on the indexers and in the search app on our search heads; all with no luck. I have also tried various values in transforms.conf ranging from both MetaData:Source and source for SOURCE_KEY and using source:: in the regex. Anyone have any ideas what I'm doing wrong?

Here is what is currently in transforms.conf and props.conf:
transforms.conf:

[set-host-elasticsearch]
SOURCE_KEY = MetaData:Source
REGEX = /hosts/([^/]+)/logs/
DEST_KEY = MetaData:Host
FORMAT = host::$1

props.conf:

[elasticsearch]
TRANSFORMS-set-host-elasticsearch = set-host-elasticsearch

The regex works fine in a search, I can run the following and get a table of hosts:

sourcetype=elasticsearch | rex field=source "/hosts/(?<hostname>[^/]+)/logs/" | stats count by hostname

As an example, my source path might be something like:
/rel/ps/applications/elasticsearch/hosts/tvrap571/logs/test_cre_gld.log

Re: Host override issues

lguinn2 — Mon, 03 Dec 2012 19:43:57 GMT

What is currently in your props.conf and transforms.conf should work - these files should be on all of your indexers.

However, you could also do this more easily on the forwarder using inputs.conf. This solution is also much more efficient.

[monitor://yourinputhere]
host_regex = /hosts/(?<hostname>[^/]+)/logs/

More info here

Re: Host override issues

wwwdrich — Tue, 04 Dec 2012 15:38:45 GMT

These are universal forwarders, I thought they didn't support any manipulation of the data? I'll have to give that a try, since the forwarder configs are managed via. the deployment server it's an easy fix. I'll add a comment with how it goes.

Thanks!

Re: Host override issues

wwwdrich — Tue, 04 Dec 2012 17:20:15 GMT

Great news, that works!! Thanks for the tip, I didn't think it would honor that on the forwarders.

Re: Host override issues

lguinn2 — Tue, 04 Dec 2012 21:29:31 GMT

Here's the deal - a Universal Forwarder (UF) does not do parsing

However, if you assign a sourcetype or host to the entire data stream in inputs.conf, that works. Because the UF doesn't have to parse anything to do it. You can even set values in props.conf on the UF - as long as you are referring to properties that apply at input time and require no parsing. For example, you can override the automatic sourcetype of a source in props.conf on the UF.