Getting Data In
Highlighted

Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Communicator

As part of some custom requirement we need to forward the same data to two different indexes on same physical indexer.

Is there a way the Universal forwarder can send the data to two different indexes?

In my inputs.conf of forwarder, I tried with below configuration:

[monitor:///data/logs/test.log]
index=index-1
sourcetype = myfirstsource

[monitor:///data/logs/test.log]
index=index-2
sourcetype = myfirstsource

it looks like the same cannot be achievable using a Splunk UF. Is there a way the same can be configured? Any help would be appreciated ..

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

SplunkTrust
SplunkTrust

Hi,

You can do this, but probably not the way you think.

You can ask Splunk (UF or full instances) to clone every data to multiple indexers using pools and configuration in outputs.conf, but this will not answer your need.

Specifically for your testing case, I would install and run 2 independant UF, having each one inputs watching for your files, forwarding data to same indexer at the difference of the destination index name, and this will do the trick.

In any way, you cannot have one the same stanza twice in the same instance, only one will be considered by Splunk.
Another way could be playing with symbolic links (assuming you are using a real OS, not some Win* things)
adding crcsalt directive will ensure Splunk considers files (adding path for checksum evaluation)

To resume, 2 different stanza in inputs.conf with crcsalt pointing to same real files, but at least one monitor accessing files with a sym link. (And the other directly)
Not tested, but it could work

The dual instance will work for sure.

Hope this helps

Guilhem

Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Communicator

Thanks for your input which is useful ,It looks like we have some limitation here .

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Motivator

As mentioned an option is to run 2 forwarders. The biggest issue (for most folks I'd think) is the data would count against your license twice. One thing you could do is bring the data into one index and then use the | collect command to make a copy and put it into the second index. Something like

index=foo sourcetype=bar | collect index=baz sourcetype=bar

You don't have to set the sourcetype but then it will be set to "stash" and won't have any field extractions. I'm somewhat curious at the use case as you might also be able to get away with something like summary searches if the root thing is something like user permissions to the data.

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

SplunkTrust
SplunkTrust

Runals is right, Summary indexing can be a way to answer your request:

http://docs.splunk.com/Documentation/Splunk/6.2.5/Knowledge/Usesummaryindexing

At the advantage of non supplementary licencing cost

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Communicator

Thanks Runal, Gulimxm,
The reason I was thinking of different index, as we have some requirement to send the data to a new indexer for some of the hosts and one particular sourcetype.

Looks like summary index will not help here..

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Motivator

In the end though are you wanting the data to exist in both indexes or just one? If it needs to live in Splunk twice are you willing to have the data count twice against your license or no? I have a number of forwarders on systems that are sending data to different indexes. Depending on the complexity of the data and how/where it is being ingested that can be controlled on the inputs side and some on the indexing side.

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

Communicator

Hi Runal,

At the end we just want to send the data to only one indexer. To give you more clarity , currently the data is getting indexed in os index and we need the data to be moved to a different index before sending it to another splunk enterprise indexer ( some other system as part of process , can't have same index) . So my question is ..

  1. Can I send the data only for one sourcetype, and index=os and for few hosts to different splunk enterprise index, by changing the index name using transforms and props ?

I know that the index name can be overridden by the splunk enterprise index receiving the data, but have some process limitation ..

0 Karma
Highlighted

Re: Is it possible to configure a universal forwarder to send data to two different indexes on the same indexer?

SplunkTrust
SplunkTrust

Yes you can totally.

I think your question was not clear to us.

You can use a rex expression in a transform that will rewrite medata, such as the index name destination.

For example:

In transforms.conf you would use:

[override-index]
SOURCE_KEY = _raw
REGEX = (mymatch).*
FORMAT = my_new_index
DEST_KEY = _MetaData:Index

In your props.conf you will associate

[source::mysource]
TRANSFORMS-sourcetype = override-index

http://docs.splunk.com/Documentation/Splunk/6.2.5/admin/Transformsconf

--> There are multiple possibility for source:: definition depending where comes from your data, look here:

http://docs.splunk.com/Documentation/Splunk/6.2.5/Admin/propsconf

So that part of the data you will be receiving from one input (your UF monitor) will be sent to a different index.

Other meatadata can be rewritten too, such as the sourcetype

Note these parsing operation (rewriting metadata) will occurs on indexer (this is indexing time parsing), not on the Universal Forwarder.
That configuration will be part of indexers, or Heavy Forwarders if UF are forwarding data to them before indexers.

0 Karma