topic Re: Have forwarder duplicating data to 2 indexes. in Getting Data In

Have forwarder duplicating data to 2 indexes.

mataharry — Tue, 09 Oct 2012 18:42:04 GMT

For some inputs on a forwarder, I want to send the same data to the same indexer, but duplicate it in 2 indexes (they have different permissions/retention)

This is sending to the main index, I want to main and public.
[monitor:///var/log/feed] disabled = false followTail = 0 sourcetype = one index=main

Re: Have forwarder duplicating data to 2 indexes.

mataharry — Tue, 09 Oct 2012 18:42:53 GMT

I tested with multiple groups in outputs.conf but I cannot change the index and they all go to the same index.

Re: Have forwarder duplicating data to 2 indexes.

yannK — Tue, 09 Oct 2012 18:45:57 GMT

2 methods :
A - Use another instance of splunk monitoring the same file and specifying a different index
(by example on windows you can have 1 UF and 1 LMF/HF/indexer, on linux, as many instances as you want)

B - use a symlink to the files/folders and have a secondary monitor on the symlink (with a different index destination)
see screenshot for the result

create a symlink
example :
ln -s feed symlink

and define 2 inputs one on the original, the other on the symlink

[monitor:///var/log/feed] 
disabled = false 
followTail = 0 
sourcetype = one 
index=main 

[monitor:///var/log/symlink] 
disabled = false 
followTail = 0 
sourcetype = one
 # or any other sourcetype 
crcSalt=< SOURCE >
 # required to force splunk to differentiate files based on the path/filename, write SOURCE in caps (the html formatting may hide it), and remoce the space in the tag.
index=public
 # the index of your choice 
followSymlink=true 
 # to make sure that the symlink will be followed.