How can I index the same file to different indexes?

[Simeon]

I have a file that I need to index twice. Specifically, I need it sent/indexed to two different indexes. How could I have a single file get indexed to two separate indexes?

[Simeon]

Let us consider that you have a very common log file (/var/log/messages) that gets indexed to an index called operations. Your environment has hundreds of machines that send this information to Splunk via a common forwarder configuration. Now, what if I have a few machines where I want this log to be shared with another group that should not have access to the operations index. Well, I could create a role filter for this other group and grant access to this index. For my case, I don't want multiple filters and would like to silo my data.

[jrodman]

There's a few use cases: apps that overlap in the data they want but are developed entirely independently; multi-tenant environments where you want different splunk groups to operate relatively independently; a case similar to summary indexing where you want sparse data in a special index, but you also want it available in a general, easily accessible index.

[Simeon · Accepted solution]

You can use the combination of crcSalt and a symlink. Consider the following example:

I have two indexes, index1 and index2. The file I want to index is called error_log and resides in /var/log/httpd/.

By symlinking log_file to another location, and using crcSalt, Splunk will be able to index this file twice and send it to another index. First, symbolically link the error_log file to another location:

>cd /var/log/httpd/dir
>ln -s /var/log/httpd/error_log .

The previous commands will create a soft link pointing to /var/log/httpd/dir/error_log. Here is the exact configuration in inputs.conf:

[monitor:///var/log/httpd/error_log]
index=index1
sourcetype=apache_error
crcSalt = <SOURCE>

[monitor:///var/log/httpd/dir/error_log]
index=index2
sourcetype=apache_error
crcSalt = <SOURCE>

[hulahoop]

I'm interested in understanding why you would like to do this. 🙂