topic Re: How can I index the same file to different indexes? in Getting Data In

How can I index the same file to different indexes?

Simeon — Wed, 14 Apr 2010 05:50:31 GMT

I have a file that I need to index twice. Specifically, I need it sent/indexed to two different indexes. How could I have a single file get indexed to two separate indexes?

Re: How can I index the same file to different indexes?

hulahoop — Wed, 14 Apr 2010 05:53:52 GMT

I'm interested in understanding why you would like to do this. 🙂

Re: How can I index the same file to different indexes?

Simeon — Wed, 14 Apr 2010 05:59:27 GMT

You can use the combination of crcSalt and a symlink. Consider the following example:

I have two indexes, index1 and index2. The file I want to index is called error_log and resides in /var/log/httpd/.

By symlinking log_file to another location, and using crcSalt, Splunk will be able to index this file twice and send it to another index. First, symbolically link the error_log file to another location:

>cd /var/log/httpd/dir
>ln -s /var/log/httpd/error_log .

The previous commands will create a soft link pointing to /var/log/httpd/dir/error_log. Here is the exact configuration in inputs.conf:

[monitor:///var/log/httpd/error_log]
index=index1
sourcetype=apache_error
crcSalt = <SOURCE>

[monitor:///var/log/httpd/dir/error_log]
index=index2
sourcetype=apache_error
crcSalt = <SOURCE>

Re: How can I index the same file to different indexes?

jrodman — Wed, 14 Apr 2010 07:25:47 GMT

There's a few use cases: apps that overlap in the data they want but are developed entirely independently; multi-tenant environments where you want different splunk groups to operate relatively independently; a case similar to summary indexing where you want sparse data in a special index, but you also want it available in a general, easily accessible index.

Re: How can I index the same file to different indexes?

Simeon — Thu, 15 Apr 2010 00:21:20 GMT

Let us consider that you have a very common log file (/var/log/messages) that gets indexed to an index called operations. Your environment has hundreds of machines that send this information to Splunk via a common forwarder configuration. Now, what if I have a few machines where I want this log to be shared with another group that should not have access to the operations index. Well, I could create a role filter for this other group and grant access to this index. For my case, I don't want multiple filters and would like to silo my data.