Knowledge Management

Routing summary indexes to specific indexer based on summary index name

melonman
Motivator

Hi,

Could anyone help me with configuration for the following?

  • summary indexes created on search head layer to indexer layer (outputs.conf)
  • send summary1 to indexer1 and 2 (clone)
  • send summary2 to indexer3 and 4 (clone)
  • send _internal index to indexer1,2,3,4 (spray)

Actual configuration will be more complicated, but I would like to know how to do this as an example of summary index routing.

Any comment would be really appreciated.

0 Karma

melonman
Motivator

The following setting worked, but I am still not sure about blockOnCloning and some other important parameters for cloning in outputs.conf can work for _TCP_ROUTING in transforms.conf...

Anyway, this is what I've got sp far.

outputs.conf

[tcpout]
defaultGroup = sprayAll

[tcpout:sprayAll]
server = 127.0.0.1:19997,127.0.0.1:29997,127.0.0.1:39997,127.0.0.1:49997
autoLB = true
autoLBFrequency = 13


[tcpout:idx1_9997]
server = 127.0.0.1:19997

[tcpout:idx2_9997]
server = 127.0.0.1:29997

[tcpout:idx3_9997]
server = 127.0.0.1:39997

[tcpout:idx4_9997]
server = 127.0.0.1:49997

props.conf

[stash_new]
TRANSFORMS-routing = summary1,summary2

transforms.conf

[summary1]
SOURCE_KEY = _MetaData:Index
REGEX = summary1
DEST_KEY = _TCP_ROUTING
FORMAT = idx1_9997,idx2_9997

[summary2]
SOURCE_KEY = _MetaData:Index
REGEX = summary2
DEST_KEY = _TCP_ROUTING
FORMAT = idx3_9997,idx4_9997

guitarmansevevn
Engager

This did the trick for us. Note that all the other summary indexing will use the default routing. This is exactly what we needed to happen.

0 Karma

somesoni2
Revered Legend

See this link for details on to selectively forward data from an index.

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Filter_data_by_tar...

I believe something like this should work
outputs.conf on Search Head

[tcpout:indexer1]
server=server1:9997
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
forwardedindex.0.whitelist = summary1
forwardedindex.1.whitelist = _internal

[tcpout:indexer2]
server=server1:9997
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
forwardedindex.0.whitelist = summary1
forwardedindex.1.whitelist = _internal

[tcpout:indexer3]
server=server1:9997
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
forwardedindex.0.whitelist = summary2
forwardedindex.1.whitelist = _internal

[tcpout:indexer4]
server=server1:9997
#Disable the current filters from the defaults outputs.conf
forwardedindex.0.whitelist = 
forwardedindex.1.blacklist =
forwardedindex.2.whitelist =
forwardedindex.0.whitelist = summary2
forwardedindex.1.whitelist = _internal
0 Karma

melonman
Motivator

I thought the same thing, but actually the filter is only applicable to [tcpout] stanza, as stated in outputs.conf.spec.

#----Index Filter Settings.
# These attributes are only applicable under the global [tcpout] stanza.
# This filter does not work if it is created under any other stanza.
forwardedindex.<n>.whitelist = <regex>
forwardedindex.<n>.blacklist = <regex>

Probably what Splunk can do with this configuration is to select which index to be fotwarded or not. and this is not for selecting a destinating indexers... maybe.

0 Karma
Get Updates on the Splunk Community!

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

  Now On Demand  Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research ...

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...