Can i single file be forwarder to 2 different indexes ??


Hi Team.

I have a file name called /var/log/messages which I need to send to two different indexes (say index="A" and index="B") on 2 set of indexer servers.

i.e index="A" sourcetype="A" on the indexers set 1 - and index="B" sourcetype="B" on the indexers set 2 -- should give me the same source list i.e /var/log/messages/

please help


Tags (1)
0 Karma

Revered Legend

I believe it would involve two steps, data cloning and metadata override. Before we go further, let me remind you that it would cause double license usage for the file.

Assuming you've a data input (inputs.conf) setup to monitor the file with assigned index=indexA and sourcetype=sourcetypeA.

Step 1: Data cloning
You could configure your forwarder to clone the same data to two different indexers. See below link on corresponding outputs.conf configuration

Step 2: Metadata override
On Indexer two have following props.conf and transforms.conf


...other linebreaking/timestamp recognition settings..
TRANSFORMS-override = changeindex,changesourcetype


FORMAT = sourcetype::sourcetypeB
DEST_KEY = MetaData:Sourcetype

DEST_KEY =_MetaData:Index
 FORMAT = indexB
0 Karma


That should work if i am not wrong. Cause we do have a lot of issues where there is duplicacy in the logs getting indexed.

I would recommend you to try it out on a test environment to be cautious.

0 Karma