Getting Data In

Unable to send same source data to two different logical indexes and two different indexers groups.

Path Finder

Hi All,

Facing few challlenges, mine is playing around with the same transforms.

I'm trying to achieve the same source data to forward to two different logical indexes and two different indexers groups.

Below is my senrio.

In props.conf used

[source::DualDataTesting]
TRANSFORMS-source = Stan1, Stan2

In transforms.conf

[Stan1]
SOURCEKEY = MetaData:Source
REGEX = .
DEST
KEY = MetaData:Index
FORMAT = Index1
DEST
KEY = TCPROUTING
FORMAT = IndexerGroup1

[Stan2]
SOURCEKEY = MetaData:Source
REGEX = .
DEST
KEY = MetaData:Index
FORMAT = Index2
DEST
KEY = TCPROUTING
FORMAT = IndexerGroup2

Currently the above conf is not working.

Please any suggestion can we workaround for this ?

Thanks,
Arun Sunny

0 Karma

Motivator

try this

inputs.conf
[monitor://filepath1]
index=index1
TCPROUTING = indexergroup1

[monitor://filepath1]
index=index2
TCPROUTING = indexergroup2

Outputs.conf

[tcpout:indexergroup1]
server=server1:9997

[tcpout:indexergroup2]
server=server2:9997

0 Karma

Path Finder

Actually, I was trying for one of the DB input sources, so I cant duplicate the monitor stanza in inputs.conf

Thanks,

0 Karma

Path Finder

And I believe we can play around only once in _MetaData key values in transforms.conf .

0 Karma

Motivator

yeah thus why i have two different sourcetype for a source. But you mentioned that it is writing to only one sourcetype. May be you can try one with TCPROUTING and another with SYSLOGROUTING.

Check the below link,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Topic: Replicate a subset of data to a third-party system

0 Karma

Motivator

try this,

#props.conf
[source::DualDataTesting]
sourcetype=sourcetype1

[source::DualDataTesting]
sourcetype=sourcetype2

[sourcetype1]
TRANSFORMS-index_outputgroup1 = overrideindex1,outputgroup1

[sourcetype2]
TRANSFORMS-index_outputgroup2 = overrideindex2,outputgroup2

Transforms.conf

[overrideindex1]
DESTKEY =MetaData:Index
REGEX = .
FORMAT = mynewindex1

[overrideindex2]
DESTKEY =MetaData:Index
REGEX = .
FORMAT = mynewindex2

[outputgroup1]
REGEX=(.)
DESTKEY=TCP_ROUTING
FORMAT=outputgroup11

[outputgroup2]
REGEX=(.)
DESTKEY=TCP_ROUTING
FORMAT=outputgroup22

Outputs.conf

[tcpout:outputgroup11]
server=server1:9997

[tcpout:outputgroup22]
server=server1:9997

0 Karma

Path Finder

Its working fine for one output group and other is completely stopped sending events 😞 .

0 Karma

Motivator

Did you check data is writing on both the index and sourcetype.

0 Karma

Path Finder

Yes, I checked. It's writing only to the first index and passing the same to group1 indexers.

0 Karma

Legend

Hi arunsunny,
do you want to send all logs to:

  • both the indexers groups,
  • selectively some logs to one group, some other to another group and some logs to both the groups?

if the first, you don't need to configure props and transforms, you have only to configure outputs.conf

[tcpout:Group1]
defaultGroup = default-autolb-group

[tcpout-server://xx.xxx.xxx.xx:9997]
[tcpout-server://yy.yyy.yyy.yy:9997]

[tcpout:default-autolb-group]
server = xx.xxx.xxx.xx:9997, yy.yyy.yyy.yy:9997
disabled = false

[tcpout:Group2]
server=aa.aaa.aaa.aa:9997, bb.bbb.bbb.bb:9997
disabled = false

[tcpout-server://aa.aaa.aaa.aa:9997]
[tcpout-server://bb.bbb.bbb.bb:9997]

If the second, follow http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
In other words you have to configure an outputs.conf as above and in every inputs.conf stanza put:

  • TCPROUTING=Group1 for logs to send only to Indexers Group1
  • TCPROUTING=Group2 for logs to send only to Indexers Group2
  • nothing for logs to send to both the Indexers Groups

Bye.
Giuseppe

0 Karma