dynamic index assignment based on event or log prefix

[dhavamanis]

We do have the sample event like below and these event logs are coming from syslog-ng forwarder. Can you please tell us, how to assign the index based on event prefix value.

Example Event :

nbcutve bal-8080 1.2.3.4 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo bal-8079 1.2.3.5 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo bal-6339 1.2.3.6 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo bal-6339 1.2.3.7 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcutve bal-8079 1.2.3.4 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"

In the above example, nbcutve and nbcubravo are the different brand, how to assign different brand name to appropriate index name (nbcutve / nbcubravo ) dynamically in transform.conf and inputs.conf. Can you please share the config details.

Another Example logs : (please provide REGEX for the below pattern prefix extraction as well)

nbcutve.varnishncsa bal-8080 1.2.3.4 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo.varnishncsa bal-8079 1.2.3.5 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo.varnishncsa bal-6339 1.2.3.6 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcubravo.varnishncsa bal-6339 1.2.3.7 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"
nbcutve.varnishncsa bal-8079 1.2.3.4 - - [11/Aug/2014:15:10:04 +0000] "GET http://test.com HTTP/1.1"

[somesoni2 · Accepted solution]

Try something like this

On Indexer
props.conf

[YourSourceType]
..
..
TRANSFORMS-idx_assign = assign_idx


transforms.conf

[assign_idx]
DEST_KEY = _MetaData:Index
REGEX = ^(\w*)(\s|\.)
FORMAT = $1

Need a restart after updated the conf files