Splunk Dev

Sending data between dev and prod indexers

jiaqya
Builder

i have a dev and prod setup.
We cannot have UF agent installed on splunk infra servers , as splunk does not support it.
so we have setup a way to collect capacity/cpu/mem data just like uf agent for our splunk servers.
now we have production server data in the production indexers and dev server data on dev indexers.
but we are showing it on a report that is there on production.

so we have a situation to send the dev indexers data to production indexers( index=test) for showing the capacity data for development also on production report.

what is the best way to send selective index (index=test) from dev indexer to production indexers( index=test) so that our production report can see both the data.

Tags (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

Instead I would make prod search heads search both dev and production indexers.

That's much easier than copying data between environments etc.

See distsearch.conf and mind that you need to connect to cluster masters instead of directly to peers when in a clustered environment.

0 Karma

jiaqya
Builder

basically how to get 1 dev index data into 1 prod index without changing the configuration or with minimal change..

0 Karma

nickhills
Ultra Champion

I am just re-reading your question. What do you mean:

We cannot have UF agent installed on splunk infra servers , as splunk does not support it.

You can install a UF on a Splunk server. You just need to configure it to startup with a different management port.

If I understand your requirements, you want to capture the logs from your development Splunk Infrastructure, (I am guessing using the ta-nix app for OS logs and metrics?) but send those logs to your production Splunk cluster.

You absolutely can do that with a UF installed on your Splunk servers, and it is supported.
You can make the change in system/local/web.conf on the UI

mgmtHostPort = <IP address:port>
* The IP address and host port of the splunkd process.
* Don't include "http[s]://" when specifying this setting. Only 
  include the IP address and port.
* Default: 0.0.0.0:8089

Or set it on the command line when you start the UF the first time

If my comment helps, please give it a thumbs up!
0 Karma

jiaqya
Builder

We have tried the port number fix you mentioned and we had a case with splunk also, and splunk told us that it is not supported to have splunk uf installed on Splunk infra servers.

is there any other way to do this ?

like, running a saved search on development and pointing to a summary index which is on production indexer. something like this. is there any such thing we can try...

0 Karma

nickhills
Ultra Champion

splunk told us that it is not supported to have splunk uf installed on Splunk infra servers

Interesting.. I have had the opposite advice from support for one of my clients, but it was a specific use case.
The only reference I can see cautioning against it is on windows.

As an alternative you can use the outputs.conf on your dev indexers to specify an alternative tcpout group.
see: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

You then use props.conf and transforms.conf to selectively route data to the additional output group.
If you have an open dialogue with Splunk support they should be able to help you with this.

If my comment helps, please give it a thumbs up!
0 Karma

jiaqya
Builder

This looks promising, will try this and get back.. thanks..

0 Karma

nickhills
Ultra Champion

Do you need to do this for all historic data, or just new data?

If my comment helps, please give it a thumbs up!
0 Karma

jiaqya
Builder

any New data is fine, historic is preferred but not mandatory..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...