topic Re: Sending data between dev and prod indexers in Splunk Dev

Sending data between dev and prod indexers

jiaqya — Thu, 27 Feb 2020 06:25:44 GMT

i have a dev and prod setup.
We cannot have UF agent installed on splunk infra servers , as splunk does not support it.
so we have setup a way to collect capacity/cpu/mem data just like uf agent for our splunk servers.
now we have production server data in the production indexers and dev server data on dev indexers.
but we are showing it on a report that is there on production.

so we have a situation to send the dev indexers data to production indexers( index=test) for showing the capacity data for development also on production report.

what is the best way to send selective index (index=test) from dev indexer to production indexers( index=test) so that our production report can see both the data.

Re: Sending data between dev and prod indexers

jiaqya — Thu, 27 Feb 2020 06:31:29 GMT

basically how to get 1 dev index data into 1 prod index without changing the configuration or with minimal change..

Re: Sending data between dev and prod indexers

nickhills — Thu, 27 Feb 2020 09:35:41 GMT

Do you need to do this for all historic data, or just new data?

Re: Sending data between dev and prod indexers

jiaqya — Thu, 27 Feb 2020 10:14:40 GMT

any New data is fine, historic is preferred but not mandatory..

Re: Sending data between dev and prod indexers

nickhills — Thu, 27 Feb 2020 10:42:44 GMT

I am just re-reading your question. What do you mean:

We cannot have UF agent installed on splunk infra servers , as splunk does not support it.

You can install a UF on a Splunk server. You just need to configure it to startup with a different management port.

If I understand your requirements, you want to capture the logs from your development Splunk Infrastructure, (I am guessing using the ta-nix app for OS logs and metrics?) but send those logs to your production Splunk cluster.

You absolutely can do that with a UF installed on your Splunk servers, and it is supported.
You can make the change in system/local/web.conf on the UI

mgmtHostPort = <IP address:port>
* The IP address and host port of the splunkd process.
* Don't include "http[s]://" when specifying this setting. Only 
  include the IP address and port.
* Default: 0.0.0.0:8089

Or set it on the command line when you start the UF the first time

Re: Sending data between dev and prod indexers

jiaqya — Thu, 27 Feb 2020 11:04:51 GMT

We have tried the port number fix you mentioned and we had a case with splunk also, and splunk told us that it is not supported to have splunk uf installed on Splunk infra servers.

is there any other way to do this ?

like, running a saved search on development and pointing to a summary index which is on production indexer. something like this. is there any such thing we can try...

Re: Sending data between dev and prod indexers

nickhills — Thu, 27 Feb 2020 11:11:59 GMT

splunk told us that it is not supported to have splunk uf installed on Splunk infra servers

Interesting.. I have had the opposite advice from support for one of my clients, but it was a specific use case.
The only reference I can see cautioning against it is on windows.

As an alternative you can use the outputs.conf on your dev indexers to specify an alternative tcpout group.
see: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

You then use props.conf and transforms.conf to selectively route data to the additional output group.
If you have an open dialogue with Splunk support they should be able to help you with this.

Re: Sending data between dev and prod indexers

jiaqya — Thu, 27 Feb 2020 11:18:02 GMT

This looks promising, will try this and get back.. thanks..

Re: Sending data between dev and prod indexers

jkat54 — Thu, 27 Feb 2020 11:52:32 GMT

Instead I would make prod search heads search both dev and production indexers.

That's much easier than copying data between environments etc.

See distsearch.conf and mind that you need to connect to cluster masters instead of directly to peers when in a clustered environment.