Unable to send same source data to two different logical indexes and two different indexers groups.

[arunsunny]

Hi All,

Facing few challlenges, mine is playing around with the same transforms.

I'm trying to achieve the same source data to forward to two different logical indexes and two different indexers groups.

Below is my senrio.

In props.conf used

[source::Dual_Data_Testing]
TRANSFORMS-source = Stan1, Stan2

In transforms.conf

[Stan1]
SOURCE_KEY = MetaData:Source
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = Index1
DEST_KEY = _TCP_ROUTING
FORMAT = IndexerGroup1

[Stan2]
SOURCE_KEY = MetaData:Source
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = Index2
DEST_KEY = _TCP_ROUTING
FORMAT = IndexerGroup2

Currently the above conf is not working.

Please any suggestion can we workaround for this ?

Thanks,
Arun Sunny

[sbbadri]

try this

inputs.conf
[monitor://filepath1]
index=index1
_TCP_ROUTING = indexergroup1

[monitor://filepath1]
index=index2
_TCP_ROUTING = indexergroup2

Outputs.conf

[tcpout:indexergroup1]
server=server1:9997

[tcpout:indexergroup2]
server=server2:9997

[arunsunny]

Actually, I was trying for one of the DB input sources, so I cant duplicate the monitor stanza in inputs.conf

Thanks,

[arunsunny]

And I believe we can play around only once in _MetaData key values in transforms.conf .

[sbbadri]

yeah thus why i have two different sourcetype for a source. But you mentioned that it is writing to only one sourcetype. May be you can try one with _TCP_ROUTING and another with _SYSLOG_ROUTING.

Check the below link,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Topic: Replicate a subset of data to a third-party system

[sbbadri]

try this,

#props.conf
[source::Dual_Data_Testing]
sourcetype=sourcetype1

[source::Dual_Data_Testing]
sourcetype=sourcetype2

[sourcetype1]
TRANSFORMS-index_outputgroup1 = overrideindex1,outputgroup1

[sourcetype2]
TRANSFORMS-index_outputgroup2 = overrideindex2,outputgroup2

Transforms.conf

[overrideindex1]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index1

[overrideindex2]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index2

[outputgroup1]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=outputgroup11

[outputgroup2]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=outputgroup22

Outputs.conf

[tcpout:outputgroup11]
server=server1:9997

[tcpout:outputgroup22]
server=server1:9997

[arunsunny]

Its working fine for one output group and other is completely stopped sending events 😞 .

[sbbadri]

Did you check data is writing on both the index and sourcetype.

[arunsunny]

Yes, I checked. It's writing only to the first index and passing the same to group1 indexers.

[gcusello]

Hi arunsunny,
do you want to send all logs to:

• both the indexers groups,
• selectively some logs to one group, some other to another group and some logs to both the groups?

if the first, you don't need to configure props and transforms, you have only to configure outputs.conf

[tcpout:Group1]
defaultGroup = default-autolb-group

[tcpout-server://xx.xxx.xxx.xx:9997]
[tcpout-server://yy.yyy.yyy.yy:9997]

[tcpout:default-autolb-group]
server = xx.xxx.xxx.xx:9997, yy.yyy.yyy.yy:9997
disabled = false

[tcpout:Group2]
server=aa.aaa.aaa.aa:9997, bb.bbb.bbb.bb:9997
disabled = false

[tcpout-server://aa.aaa.aaa.aa:9997]
[tcpout-server://bb.bbb.bbb.bb:9997]

If the second, follow http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
In other words you have to configure an outputs.conf as above and in every inputs.conf stanza put:

• _TCP_ROUTING=Group1 for logs to send only to Indexers Group1
• _TCP_ROUTING=Group2 for logs to send only to Indexers Group2
• nothing for logs to send to both the Indexers Groups

Bye.
Giuseppe