Archive
Highlighted

route data from multiple universal fwders passsing through heavy ones

Explorer

Hello,

we are planning to have multiple Universal Fwders, forwarding data to 2 Load Balanced Heavy Fwders in cluster environment.
Doing some testing I could not manage to route data to new created index.
My scenario in detail is as follows:

1.UniversalFwder

outputs.conf

[tcpout:default-autolb-group]
server = fwd1:8888, fwd2:8888
  1. HeavyFwders

input.conf

[splunktcp://8888]

outputs.conf

[tcpout]
defaultGroup = indexer01,indexer02

[tcpout:indexer01]
server = index1:9997
useACK=true

[tcpout:indexer02]
server =index2:9997
useACK=true

props.conf

[host::universalfwder01]
TRANSFORMS-FORMAT = route-somenew-index

transforms.conf

[route-somenew-index]
SOURCE_KEY=MetaData:Host
DEST_KEY=_MetaData:Index
REGEX=^host::(:universalfwder01)$
FORMAT= somenewindex
WRITE_META=true
  1. Indexers

Indexers are receiving configurations from the master node through apply cluster-bundle

etc/master-app/_cluster/local/props.conf (propagated to both indexers into slave-apps/_cluster/local)

props.conf

[index1:9997,index2:9997]
TRANSFORMS-index = route-somenew-index

transforms.conf

[route-somenew-index]
SOURCE_KEY=MetaData:Host
DEST_KEY=_MetaData:Index
REGEX=^host::(/hostname01|hostname02*/ig)$
FORMAT= somenewindex
WRITE_META=true

Index is been created but receives no data. Where did I go wrong ?

Any help really appreciated

Tags (2)
0 Karma
Highlighted

Re: route data from multiple universal fwders passsing through heavy ones

Influencer

Are you setting the index on the heavy forwarders and the indexers for a reason?

You only need to do it once

Also you're regexes look wierd.

REGEX=^host::(:universalfwder01)$

should be

REGEX=^host::universalfwder01$

And this :

REGEX=^host::(/hostname01|hostname02*/ig)$

should probably be

REGEX=^host::(?i)(hostname01|hostname02*)$

Lasly, you dont need WRITE_META=true

Working example I use :

[set_index]
SOURCE_KEY = MetaData:Host
REGEX = ^host::\w{6}(\w{2})
DEST_KEY = _MetaData:Index
FORMAT = $1

This puts data into the relevant index based on the 7th+8th character of hostname

0 Karma
Highlighted

Re: route data from multiple universal fwders passsing through heavy ones

Explorer

Thanks a lot for your response.

Tried to put props and transforms only in HeavyFwders

transforms.conf

[route-new-index]
SOURCE_KEY=MetaData:Host
DEST_KEY=_MetaData:Index
REGEX=^host::\hostname01|hostname02\$
FORMAT= newindex

props.conf

[host::192.1678.1.2,192.168.1.3]
TRANSFORMS-FORMAT = route-new-index

Just for the record my indexes.conf is being ditributed to peers by the master node

indexes.conf

[newindex]

    homePath   = $SPLUNK_HOME/var/lib/splunk/newindex/db
    coldPath   = $SPLUNK_HOME/var/lib/splunk/newindex/colddb
    thawedPath = $SPLUNK_HOME/var/lib/splunk/newindex/thaweddb
    repFactor = auto

This configuration didn't work either. Data is being stored in the main default index database, it's not being routed to my newindex created

0 Karma