topic Re: Dynamically update index based on client hostname in Deployment Architecture

Dynamically update index based on client hostname

boomiagarg — Wed, 19 Jun 2019 17:57:56 GMT

We have on premise deployment of splunk enterprise and are using deployment server to deploy apps to UF's (or clients). We need a way to dynamically update index based on client hostname when deployment server deploys the app or update the index of the app after it is deployed to the UF. Basically, index has to be based on the client hostname.

Not sure if there is a way to achieve this or not, any help for this would be greatly appreciated. Thanks in advance!!

Re: Dynamically update index based on client hostname

FrankVl — Wed, 19 Jun 2019 18:29:04 GMT

Can you provide some more details on how the index depends on the client hostname? Is there some logic / structure to that?

One way would be to define a TRANSFORMS that overrides the index based on hostname. Another could be to create different flavors of the app, with different indexes configured and assign each to the right set of clients based on their hostname using serverclasses. And perhaps there are more options. But it would help to better understand the logic behind the hostname to index mapping to help you figure out what the best solution would be.

Re: Dynamically update index based on client hostname

boomiagarg — Wed, 19 Jun 2019 20:58:45 GMT

So my usecase is -

clientname format -
client1-acct1
client2-acct2

client1-acct1 inputs.conf sends data to both indexes
index=index1
index=index2

client2-acct2 inputs.conf sends data to both indexes-
index=index1
index=index2

Indexes available on indexers-
index1-acct1
Index1-acct2
index2-acct1
Index2-acct2

Now I want to use props and transforms to each event logged and update the indexes to -
for client1-acct1 event with index1
index=index1-acct1
index=index2-acct1

for client2-acct2 event with index1
index=index1-acct2
index=index2-acct2

Re: Dynamically update index based on client hostname

ragedsparrow — Thu, 20 Jun 2019 03:47:37 GMT

I agree with @FrankVl here. You would most likely either utilize the server classes on your deployment server to separate out the inputs based on the client hostnames like this:

Have a serverclass for client1-acct1 to only send to index1-acct1 and index2-acct1 and so on and so forth. Server classes can be set with client filters using wildcard filters as well so your clients can be filtered like this: *-acct1 would get the apps with inputs for the acct1 indexes. Then you would do the same for the acct2 hosts and indexes.

The other option would be to use a TRANSFORMS for an override based on the hostname, but in all honestly, if you can use the deployment server and server classes, I would recommend that.

For more information on server classes:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Updating/Useserverclass.conf

Re: Dynamically update index based on client hostname

FrankVl — Thu, 20 Jun 2019 07:24:52 GMT

See @ragedsparrow his comment on how to do it through server classes on the DS. If that would not work due to the number of different acct* options and indexes, then you could use a TRANSFORMS as well. Which in this case, with the variable part of the index being literally part of the hostname, can be done quite elegantly:

props.conf

[yoursourcetype]
TRANSFORMS-setindex = set_index_by_host_acct

transforms.conf

[set_index_by_host_acct]
SOURCE_KEY = MetaData:Host
REGEX = -(acct\d+)$
DEST_KEY = _MetaData:Index
FORMAT = $0-$1

This keeps the current index setting, and appends - followed by the acct* string captured from the hostname.

Note: you are mentioning sending data to 2 indexes. I don't think you can do that by simply specifying 2 index=... settings in inputs.conf. Splunk will only use one of those. If you want to clone data into 2 indexes you probably need to take a look at using CLONE_SOURCETYPE in transforms.conf. But without full understanding of why you want to do that it is hard to go into details on that part (might be a good one to cover in a separate question if you need help with that).

Re: Dynamically update index based on client hostname

boomiagarg — Thu, 20 Jun 2019 16:17:01 GMT

hi @ragedsparrow ,
Since you are recommending using serverclasses instead of transform. Wondering if there is a benefit for one over the another Or if there are any downsides to use props and transforms.

Re: Dynamically update index based on client hostname

boomiagarg — Mon, 24 Jun 2019 16:52:01 GMT

trying to understand here, so what will be the values here for $0 and $1 ? (sorry I am very new to splunk props and transforms)

Re: Dynamically update index based on client hostname

FrankVl — Tue, 25 Jun 2019 07:11:40 GMT

$0 always contains the original value of the field mentioned in DEST_KEY, $1 contains the content of the first capturing group from the REGEX.

So when you set index=index1 in inputs.conf and the host field contains client1-acct1, then $0 will be index1 and $1 will be acct1. And as such the result of the FORMAT will be that after this transform index will be index1-acct1.