How to index same source to multiple indexes

[verbal_666]

Hi there.
I have a great problem with Splunk 7.x.x.
What I need: indexing the SAME SOURCE to 2 or more indexes... NO WAY AT ALL!!!

1. The first trick is symlink, ok; it works...not always (that's another problem, for now!)

2. Using crcSalt, which, for me, should be the real tool... NOT WORKING AT ALL
   :-( (i know first 256 bytes are used for crc, by string Salt should DO the trick!!!!!! SHOULD!!!!!! DO NOT!!!!)

Example:

[monitor:///tmp/*.TXT]
index=test
sourcetype=TMP:TXT
crcSalt = one

[monitor:///tmp/*.TXT]
index=main
sourcetype=TMP:TXT_BIS
crcSalt = two

I suppose to have same /tmp/*.TXT files indexed both 1 in test 1 in main indexes...
also if 1.TXT contais "Hallo World!". NO WAY!!! 1.TXT goes to main ONLY index... test is empty!!!

Any advice to do the trick?
Thanks, in advance.

ps. I have, for now, no problem with doubling licensing space or another thing like that, I need now a technical remedy for my issue, I need 1 SOURCE TO MULTIPLE INDEXES!!! Thanks.

[verbal_666]

This, obviously, works,

TA_LOG/default/inputs.conf
[monitor:///tmp/*.TXT]
index=main
sourcetype=TMP:TXT
crcSalt = <SOURCE>
followSymlink = true

TA_LOG2/default/inputs.conf
[monitor:///tmp_ln/*.TXT]
index=test
sourcetype=TMP:TXT_BIS
crcSalt = <SOURCE>
followSymlink = true

IMHO, crcSalt should work in same stanza, multiple inputs.conf and so... IMHO it does not work!!!!!

[verbal_666]

I'll do it, quiet, by symlinking alla paths (wooof) and....amen 🙂

Thanks all...

I'm still wondering what crcSalt is for........................................................................................???!!!???!!! Bug?

[jnudell_2]

As I explained in my original response, that's not a bug. The stanza names are the same, so Splunk has to choose settings from the one that takes precedence. That's the way Splunk works with .conf file precedence.

crcSalt works. If you have two stanzas that reference the same group of files, Splunk uses crcSalt to determine if it has read a particular file previously to prevent double ingestion (as in the case if you have symlinked files/folders, by default it will NOT ingest the file if it sees it has encountered it previously).

If you're going with the symlink idea, would you mind selecting the correct answer for your request?

[verbal_666]

So, 2 stanzas in 2 inputs.conf in the same forwarder instance work with crcSalt = xxx ?
Try soon on test environment.
Thanks.

ps. nope,

TA_LOG/default/inputs.conf
[monitor:///tmp/*.TXT]
index=main
sourcetype=TMP:TXT
crcSalt = one

TA_LOG2/default/inputs.conf
[monitor:///tmp/*.TXT]
index=test
sourcetype=TMP:TXT_BIS
crcSalt = two

Results always in ingesting to "main".
In my opinion crcSalt = xxx does not work as it should.

[verbal_666]

Thanks.

Point 1, no way, i need to do it simply in Splunk Environment (distribuited) directly without cli commands or so!
Point 2, i have also 2 and sometimes more instances of forwarders, and i realized they use internal "id" to indexing data; i realized this when, for an error, a colleague installed two indentical inputs.conf in both intances... result: double indexing in same index,st 🙂 ... so, maybe, i can think using this "trick" to do my job for this personal project.

So definitively there's no way to force a salt to hash an entry log in a same forwarder instance, also in two different inputs.conf. Bad, for me, bad... crcSalt SHOULD do this work, like documentaton SAYS! Very bad for a product like Splunk.

Yes, using symlink i said before the trick is done; but it's a very frusrating job, when 1) you do not access the systems directly and works only in Deployment 2) sources and paths are.....many and more then....1....maybe 30 paths and more!!??!!??

BAD, very BAD!!!

Thanks, anyway, very kind.

[hmarkus]

You send it to the same Indexer? You could send it to one index, and clone data in Splunk 🙂

[amitm05]

Hi verbal_666

I think you can sort your problem by following the alternative here -
https://answers.splunk.com/answers/690291/one-source-to-two-indexes.html

[jnudell_2]

Hi verbal_666,
Your solution will not work because of the way Splunk reads .conf files. It uses .conf file precedence to pick only one of those two stanzas, because the stanza name is exactly the same.

There are a couple of ways to do what you're asking:
1. Run a collect command to "copy" the events from the main index to the text index
2. Install a second UF on the same box with different management ports to monitor the same file, but put it in a different index
3. Duplicate the file at the OS level and create two different stanzas in Splunk (ie; Create a symlink for the folder) <-- This is probably the best way

I was able to verify this works:
mkdir /tmp/test-ingest
ln -s /tmp/test-ingest /tmp/duplicate-ingest

Splunk inputs.conf:

[monitor:///tmp/test-ingest/*.txt]
index=test
sourcetype=TMP:TXT
crcSalt = one

[monitor:///tmp/duplicate-ingest/*.txt]
index=main
sourcetype=TMP:TXT_BIS
crcSalt = two

Create dummy data from OS:
echo Testing data to two indexes. > /tmp/test-ingest/testing.txt