How to use variables for index in inputs.conf to deploy one app to different indexes in several machines?

[tomatorage]

I'm trying to make a generic app to deploy (via the deployment server) where I can use a variable for the index in the inputs.conf file.
So I can deploy one app to several machines, but use a different index for each machine.

The preferred variable to use is a custom fact from puppet, but can also be the serverClass which you define in the serverclass.conf on the deployment server.

Any suggestions?

Thanks in advance.

[jmantor]

Can I do this with an envirnment variable?
I've had good luck defining SYSLOG_DIR in splunk-launch.conf and then referencing it in the path for a filemonitor.
This lets me have the name of the syslog node in the source and keeps my inputs.conf the same.
Can I do something similar to define an index in an inputs stanza?

[jpvlsmv]

Write your app to work with a search macro, and have Puppet put the correct value into $app/local/macros.conf

In macros.conf:
[index_for_this_env]
definition = index=%%puppet_replace_this%%

In your search, instead of index=foo eventtype=bar ..., you would have `index_for_this_env` eventtype=bar ... (backquotes around the macro name)

--Joe

[tomatorage]

I want to be able to have multiple environments with the same applications. You can use host to differentiate the environments and have all data indexed in one index. But I prefer to user more(and therefore smaller) indexes for performance. And be able to get rid of older data if an environment gets obsolete and needs to be cleaned.

for example
You can have environments A, B and C.
Apps 1 and 2 are on all environments and app 3 only on B and C.

Then I want to have the apps 1,2 and 3 on the deployment server and deploy them as mentioned above. With an automatic selection to use index A, B or C.

I hope this clarifies what I want.

[jrodman]

The performance characteristic of indexes for queries is not index size, but rather index density relative to queries. Putting the data in multiple indexes is not likely to help unless you make the data over an order of magnitude more dense relative to the searches are going to run. Meanwhile, if you were to do that, you would want that slice of data to be present on all indexers. Slicing the data up by indexer simply means you won't be able to do any horizontal scaling.

[tomatorage]

Hi jrodman,
Thanks for your reply.
I'm using data replication on the indexes, so I think horizontal scalability should not be a problem.

I'm still curious to an answer to my initial question. Just because I think it should be possible 🙂

[pradeepkumarg]

Is there a reason why you want to use different index for the same data in different hosts?