Splunk Enterprise

How to forward specific index data to a 3rd party?

rkhalu
Explorer

Hi all,

Im a noobja not a ninja. I have a Windows based Splunk Enterprise single node index running 7.0.2. I'm trying to use it as a relay of sorts, I have a dataset coming into dedicated index, and I'd like to either forward that data or mirror the data to a 3rd party. There is no requirement to keep the data in splunk otherwise. I've read around quite a bit and I'm probably close, but I can't seem to get something right - so far I've only been successful at redirecting all the data to 3rd party, not a subset of data as preferred. When I apply my settings, I no longer see data real time in my splunk environment, but I do see data at the 3rd party endpoint.

I'm confused if I can use the index itself as a heavy forwarder, I didn't find a props.conf file so I created one in C:\Program Files\Splunk\etc\system\local.

-outputs.conf-
[tcpout]
defaultGroup=nothing

[tcpout:3rdPartyDest]
server=aaa.bbb.ccc.ddd:514
type=tcp
sendCookedData=false

-props.conf-
[source]
TRANSFORMS-routing = transforms_3rdParty

-transforms.conf-
[SiteCode] (already existed)
filename = SiteCode.csv

[transforms_3rdParty]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=3rdPartyDest

I've seen reference to indexAndForward flags as well as setting the output default group to nothing, but I can't seem to get the right combo working properly and don't want to redirect our flow via trial and error anymore.
Any help is appreciated!

rkhalu
Explorer

No, but it is a MDR service in the similar vein of Rapid7.

0 Karma

nikita_p
Contributor

Hi,
If you want to clone data on your splunk and 3rd party splunk, then you can use following outputs.conf configurations

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=10.1.1.197:9997

[tcpout:indexer2]
server=10.1.1.200:9997

0 Karma

ppuru
Path Finder

What if the 3rd party is not a Splunk instance, but a system capable of processing inbound syslog from Splunk.
I know we can forward newly indexed data from Splunk to such a 3rd party system, but can we replicate historical Splunk indexed data?

0 Karma

nikita_p
Contributor

Yes, but you will have to move historical data manually by moving buckets

0 Karma

rkhalu
Explorer

I replied above, I am successfully forwarding to a separate syslog (non splunk) device but I know there are some formatting issues, so I'm not entirely confident that my implementation is forwarding true Syslog formatted data. I think if you process (cook) it then it can re-send it in a different format. my intention was to not process too much of it, and ideally I didn't need it stored on splunk, I was just using it as a means to get the data from 3rd party to the other syslog collector that's not capable of natively leveraging the API from the 3rd party itself.

I don't know about forwarding the historical data you have already, my process relies on forwarding the data it is ingesting in real time.

0 Karma

rkhalu
Explorer

This post is not stale, I have been working diligently to try and find the solution on my own...

The linked article is similar to other articles I have found and tried in the past.
I was not able to properly adapt the linked article to do what I need, and instead redirected all the logs to the device again.

I have a single splunk node/instance, which is receiving a multitude of data points into many separate indexes. There are no other heavy forwarders in the environment. I want to take data that is being ingested into a dedicated index and either replicate it, or forward it entirely (I don't care that splunk has the data) to a 3rd party device on 514. This data is not Syslog formatted, so I will be using the sendCookedData=false flag.

Ultimately, we have a need to monitor security logs from a product that is cloud based app. There is a dedicated SplunkApp to download this data from the cloud into our prem which is installed and operating properly. I need to get this data to a non-splunk SIEM, and the device can't reach out to the Cloud itself, so I am trying to use Splunk as a relay of sorts.

Any help is appreciated.

0 Karma

ppuru
Path Finder

Hi,
Did you find solution to this problem?

0 Karma

rkhalu
Explorer

I did get it working, although it's also forwarding information from other areas of splunk I don't need so it's far form a perfect implementation. As a result I am dropping unneeded data at the other side, but in my case it's not causing any harm. Hopefully between the 3 conf files below you can put together what I am doing, I redacted the content.

In this implementation, I am downloading data from a 3rd party's API via the appropriate Splunk App (getting it into Splunk wasn't the hard part), I believe I am not processing it,or storing it locally, and then forwarding it off to a separate non splunk log collector (via syslog port, but maybe not in full syslog format) for additional analysis.

Outputs.conf
[tcpout]
indexAndForward=true

[tcpout:name of 3rd party]
disabled=false
sendCookedData=false
server=OTHERSYSLOGDEVICE:514

props.conf
source::SOURCE
TRANSFORMS-routing=Transforms_Stanza

transforms.conf
[SiteCode]
filename = SiteCode.csv I THINK THIS IS IRRELEVANT, PREEXISTING

[Transforms_Stanza]
REGEX=SOURCE
DEST_KEY=TCP_ROUTING
FORMAT=*_name of 3rd party
*

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Try this and make sure you are following the order in the .conf files

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Replicate_a_subset...

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...