Getting Data In

Host override issues

wwwdrich
Explorer

I am trying to override the host field based on an element in the source path. This is data that is being forwarded from a universal forwarder and we want the host the data originally came from as the host field, not the hostname of the forwarder.

To date, I have tried modifying etc/system/local/transforms.conf and props.conf both on the indexers and in the search app on our search heads; all with no luck. I have also tried various values in transforms.conf ranging from both MetaData:Source and source for SOURCE_KEY and using source:: in the regex. Anyone have any ideas what I'm doing wrong?

Here is what is currently in transforms.conf and props.conf:
transforms.conf:

[set-host-elasticsearch]
SOURCE_KEY = MetaData:Source
REGEX = /hosts/([^/]+)/logs/
DEST_KEY = MetaData:Host
FORMAT = host::$1

props.conf:

[elasticsearch]
TRANSFORMS-set-host-elasticsearch = set-host-elasticsearch

The regex works fine in a search, I can run the following and get a table of hosts:

sourcetype=elasticsearch | rex field=source "/hosts/(?<hostname>[^/]+)/logs/" | stats count by hostname

As an example, my source path might be something like:
/rel/ps/applications/elasticsearch/hosts/tvrap571/logs/test_cre_gld.log

Tags (2)
1 Solution

lguinn2
Legend

What is currently in your props.conf and transforms.conf should work - these files should be on all of your indexers.

However, you could also do this more easily on the forwarder using inputs.conf. This solution is also much more efficient.

[monitor://yourinputhere]
host_regex = /hosts/(?<hostname>[^/]+)/logs/

More info here

View solution in original post

lguinn2
Legend

What is currently in your props.conf and transforms.conf should work - these files should be on all of your indexers.

However, you could also do this more easily on the forwarder using inputs.conf. This solution is also much more efficient.

[monitor://yourinputhere]
host_regex = /hosts/(?<hostname>[^/]+)/logs/

More info here

View solution in original post

lguinn2
Legend

Here's the deal - a Universal Forwarder (UF) does not do parsing

However, if you assign a sourcetype or host to the entire data stream in inputs.conf, that works. Because the UF doesn't have to parse anything to do it. You can even set values in props.conf on the UF - as long as you are referring to properties that apply at input time and require no parsing. For example, you can override the automatic sourcetype of a source in props.conf on the UF.

0 Karma

wwwdrich
Explorer

Great news, that works!! Thanks for the tip, I didn't think it would honor that on the forwarders.

0 Karma

wwwdrich
Explorer

These are universal forwarders, I thought they didn't support any manipulation of the data? I'll have to give that a try, since the forwarder configs are managed via. the deployment server it's an easy fix. I'll add a comment with how it goes.

Thanks!

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!