Archive

Can i single file be forwarder to 2 different indexes ??

Motivator

Hi Team.

I have a file name called /var/log/messages which I need to send to two different indexes (say index="A" and index="B") on 2 set of indexer servers.

i.e index="A" sourcetype="A" on the indexers set 1 - and index="B" sourcetype="B" on the indexers set 2 -- should give me the same source list i.e /var/log/messages/

please help

thanks.
rakesh.

Tags (1)
0 Karma

Revered Legend

I believe it would involve two steps, data cloning and metadata override. Before we go further, let me remind you that it would cause double license usage for the file.

Assuming you've a data input (inputs.conf) setup to monitor the file with assigned index=indexA and sourcetype=sourcetypeA.

Step 1: Data cloning
You could configure your forwarder to clone the same data to two different indexers. See below link on corresponding outputs.conf configuration

http://docs.splunk.com/Documentation/Forwarder/6.5.0/Forwarder/Configureforwardingwithoutputs.conf#C...

Step 2: Metadata override
On Indexer two have following props.conf and transforms.conf

props.conf

[sourcetypeA]
...other linebreaking/timestamp recognition settings..
TRANSFORMS-override = changeindex,changesourcetype

transforms.conf

[changesourcetype]
FORMAT = sourcetype::sourcetypeB
DEST_KEY = MetaData:Sourcetype
REGEX = .

[changeindex]
DEST_KEY =_MetaData:Index
REGEX = .
 FORMAT = indexB
0 Karma

Contributor

That should work if i am not wrong. Cause we do have a lot of issues where there is duplicacy in the logs getting indexed.

I would recommend you to try it out on a test environment to be cautious.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!