topic Re: Unable to send same source data to two different logical indexes and two different indexers groups. in Getting Data In

Unable to send same source data to two different logical indexes and two different indexers groups.

arunsunny — Tue, 29 Sep 2020 15:49:28 GMT

Hi All,

Facing few challlenges, mine is playing around with the same transforms.

I'm trying to achieve the same source data to forward to two different logical indexes and two different indexers groups.

Below is my senrio.

In props.conf used

[source::Dual_Data_Testing]
TRANSFORMS-source = Stan1, Stan2

In transforms.conf

[Stan1]
SOURCE_KEY = MetaData:Source
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = Index1
DEST_KEY = _TCP_ROUTING
FORMAT = IndexerGroup1

[Stan2]
SOURCE_KEY = MetaData:Source
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = Index2
DEST_KEY = _TCP_ROUTING
FORMAT = IndexerGroup2

Currently the above conf is not working.

Please any suggestion can we workaround for this ?

Thanks,
Arun Sunny

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

gcusello — Tue, 29 Sep 2020 15:49:31 GMT

Hi arunsunny,
do you want to send all logs to:

both the indexers groups,
selectively some logs to one group, some other to another group and some logs to both the groups?

if the first, you don't need to configure props and transforms, you have only to configure outputs.conf

[tcpout:Group1]
defaultGroup = default-autolb-group

[tcpout-server://xx.xxx.xxx.xx:9997]
[tcpout-server://yy.yyy.yyy.yy:9997]

[tcpout:default-autolb-group]
server = xx.xxx.xxx.xx:9997, yy.yyy.yyy.yy:9997
disabled = false

[tcpout:Group2]
server=aa.aaa.aaa.aa:9997, bb.bbb.bbb.bb:9997
disabled = false

[tcpout-server://aa.aaa.aaa.aa:9997]
[tcpout-server://bb.bbb.bbb.bb:9997]

If the second, follow http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
In other words you have to configure an outputs.conf as above and in every inputs.conf stanza put:

_TCP_ROUTING=Group1 for logs to send only to Indexers Group1
_TCP_ROUTING=Group2 for logs to send only to Indexers Group2
nothing for logs to send to both the Indexers Groups

Bye.
Giuseppe

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

sbbadri — Tue, 29 Sep 2020 15:49:33 GMT

try this

inputs.conf
[monitor://filepath1]
index=index1
_TCP_ROUTING = indexergroup1

[monitor://filepath1]
index=index2
_TCP_ROUTING = indexergroup2

Outputs.conf

[tcpout:indexergroup1]
server=server1:9997

[tcpout:indexergroup2]
server=server2:9997

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

arunsunny — Wed, 20 Sep 2017 16:28:28 GMT

Actually, I was trying for one of the DB input sources, so I cant duplicate the monitor stanza in inputs.conf

Thanks,

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

sbbadri — Tue, 29 Sep 2020 15:49:50 GMT

try this,

#props.conf
[source::Dual_Data_Testing]
sourcetype=sourcetype1

[source::Dual_Data_Testing]
sourcetype=sourcetype2

[sourcetype1]
TRANSFORMS-index_outputgroup1 = overrideindex1,outputgroup1

[sourcetype2]
TRANSFORMS-index_outputgroup2 = overrideindex2,outputgroup2

Transforms.conf

[overrideindex1]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index1

[overrideindex2]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index2

[outputgroup1]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=outputgroup11

[outputgroup2]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=outputgroup22

Outputs.conf

[tcpout:outputgroup11]
server=server1:9997

[tcpout:outputgroup22]
server=server1:9997

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

arunsunny — Thu, 21 Sep 2017 08:57:13 GMT

Its working fine for one output group and other is completely stopped sending events 😞 .

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

sbbadri — Thu, 21 Sep 2017 14:47:44 GMT

Did you check data is writing on both the index and sourcetype.

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

arunsunny — Thu, 21 Sep 2017 14:53:27 GMT

Yes, I checked. It's writing only to the first index and passing the same to group1 indexers.

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

arunsunny — Thu, 21 Sep 2017 16:04:22 GMT

And I believe we can play around only once in _MetaData key values in transforms.conf .

Re: Unable to send same source data to two different logical indexes and two different indexers groups.

sbbadri — Tue, 29 Sep 2020 15:50:35 GMT

yeah thus why i have two different sourcetype for a source. But you mentioned that it is writing to only one sourcetype. May be you can try one with _TCP_ROUTING and another with _SYSLOG_ROUTING.

Check the below link,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Topic: Replicate a subset of data to a third-party system