topic Re: How to forward specific index data to a 3rd party? in Splunk Enterprise

How to forward specific index data to a 3rd party?

rkhalu — Thu, 10 May 2018 17:10:43 GMT

Hi all,

Im a noobja not a ninja. I have a Windows based Splunk Enterprise single node index running 7.0.2. I'm trying to use it as a relay of sorts, I have a dataset coming into dedicated index, and I'd like to either forward that data or mirror the data to a 3rd party. There is no requirement to keep the data in splunk otherwise. I've read around quite a bit and I'm probably close, but I can't seem to get something right - so far I've only been successful at redirecting all the data to 3rd party, not a subset of data as preferred. When I apply my settings, I no longer see data real time in my splunk environment, but I do see data at the 3rd party endpoint.

I'm confused if I can use the index itself as a heavy forwarder, I didn't find a props.conf file so I created one in C:\Program Files\Splunk\etc\system\local.

-outputs.conf-
[tcpout]
defaultGroup=nothing

[tcpout:3rdPartyDest]
server=aaa.bbb.ccc.ddd:514
type=tcp
sendCookedData=false

-props.conf-
[source]
TRANSFORMS-routing = transforms_3rdParty

-transforms.conf-
[SiteCode] (already existed)
filename = SiteCode.csv

[transforms_3rdParty]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=3rdPartyDest

I've seen reference to indexAndForward flags as well as setting the output default group to nothing, but I can't seem to get the right combo working properly and don't want to redirect our flow via trial and error anymore.
Any help is appreciated!

Re: How to forward specific index data to a 3rd party?

renjith_nair — Fri, 11 May 2018 04:03:39 GMT

Try this and make sure you are following the order in the .conf files

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Replicate_a_subset_of_data_to_a_third-party_system

Re: How to forward specific index data to a 3rd party?

rkhalu — Fri, 25 May 2018 15:11:55 GMT

This post is not stale, I have been working diligently to try and find the solution on my own...

The linked article is similar to other articles I have found and tried in the past.
I was not able to properly adapt the linked article to do what I need, and instead redirected all the logs to the device again.

I have a single splunk node/instance, which is receiving a multitude of data points into many separate indexes. There are no other heavy forwarders in the environment. I want to take data that is being ingested into a dedicated index and either replicate it, or forward it entirely (I don't care that splunk has the data) to a 3rd party device on 514. This data is not Syslog formatted, so I will be using the sendCookedData=false flag.

Ultimately, we have a need to monitor security logs from a product that is cloud based app. There is a dedicated SplunkApp to download this data from the cloud into our prem which is installed and operating properly. I need to get this data to a non-splunk SIEM, and the device can't reach out to the Cloud itself, so I am trying to use Splunk as a relay of sorts.

Any help is appreciated.

Re: How to forward specific index data to a 3rd party?

ppuru — Tue, 15 Jan 2019 12:01:25 GMT

Hi,
Did you find solution to this problem?

Re: How to forward specific index data to a 3rd party?

nikita_p — Tue, 15 Jan 2019 12:34:50 GMT

Hi,
If you want to clone data on your splunk and 3rd party splunk, then you can use following outputs.conf configurations

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9997

Re: How to forward specific index data to a 3rd party?

ppuru — Tue, 15 Jan 2019 12:53:10 GMT

What if the 3rd party is not a Splunk instance, but a system capable of processing inbound syslog from Splunk.
I know we can forward newly indexed data from Splunk to such a 3rd party system, but can we replicate historical Splunk indexed data?

Re: How to forward specific index data to a 3rd party?

rkhalu — Tue, 29 Sep 2020 22:46:39 GMT

I did get it working, although it's also forwarding information from other areas of splunk I don't need so it's far form a perfect implementation. As a result I am dropping unneeded data at the other side, but in my case it's not causing any harm. Hopefully between the 3 conf files below you can put together what I am doing, I redacted the content.

In this implementation, I am downloading data from a 3rd party's API via the appropriate Splunk App (getting it into Splunk wasn't the hard part), I believe I am not processing it,or storing it locally, and then forwarding it off to a separate non splunk log collector (via syslog port, but maybe not in full syslog format) for additional analysis.

Outputs.conf
[tcpout]
indexAndForward=true

[tcpout:name of 3rd party]
disabled=false
sendCookedData=false
server=OTHERSYSLOGDEVICE:514

props.conf
source::SOURCE
TRANSFORMS-routing=Transforms_Stanza

transforms.conf
[SiteCode]
filename = SiteCode.csv I THINK THIS IS IRRELEVANT, PREEXISTING

[Transforms_Stanza]
REGEX=SOURCE
DEST_KEY=TCP_ROUTING
FORMAT=*_name of 3rd party*

Re: How to forward specific index data to a 3rd party?

rkhalu — Tue, 15 Jan 2019 22:28:58 GMT

I replied above, I am successfully forwarding to a separate syslog (non splunk) device but I know there are some formatting issues, so I'm not entirely confident that my implementation is forwarding true Syslog formatted data. I think if you process (cook) it then it can re-send it in a different format. my intention was to not process too much of it, and ideally I didn't need it stored on splunk, I was just using it as a means to get the data from 3rd party to the other syslog collector that's not capable of natively leveraging the API from the 3rd party itself.

I don't know about forwarding the historical data you have already, my process relies on forwarding the data it is ingesting in real time.

Re: How to forward specific index data to a 3rd party?

nikita_p — Wed, 16 Jan 2019 06:42:51 GMT

Yes, but you will have to move historical data manually by moving buckets

Re: How to forward specific index data to a 3rd party?

rkhalu — Tue, 26 Feb 2019 15:41:09 GMT

No, but it is a MDR service in the similar vein of Rapid7.