All Apps and Add-ons

Is it possible to selectively forward data to a third-party application, e.g. Rapid7 InsightIDR?

nick405060
Motivator

Hi there,

Has anyone been able to send select data to InsightIDR? Their documentation is currently lacking. If so, could you share your props.conf/inputs.conf/transforms.conf or any other insights that you have?

https://insightidr.help.rapid7.com/docs/splunk

1 Solution

nick405060
Motivator

Okay so this post is going to detail what actually got it working for me. I'll leave my earlier answer and comments up for various reasons. If you want to forward all your data to the third party system, then you can use transforms/outputs/props per the third-party-system Splunk documentation or per the route-and-filter-data Splunk documentation. Otherwise this answer is the only way that I know of to selectively forward data without interfering with Splunk data collection, at least with Splunk 7.2.0.

IDR:

Event Source: Rapid7 Raw Data
Display Name: Splunk
Collection Method: Log Aggregator
Log Aggregator: Splunk
Port: PORT
Protocol: TCP (not encrypted)

Directories:

/opt/splunk/etc/system/local - No outputs or props or transforms
/opt/splunk/etc/system/default - The three forwardedindex stanzas must be commented out in outputs.conf
/opt/splunk/etc/apps/rapid7_idr - The only file here is outputs.

outputs.conf:

[tcpout:rapidreader]
server = IP:PORT
sendCookedData = false

[tcpout]
defaultGroup = rapidreader
indexAndForward = true
forwardedindex.0.blacklist = ^((?!alerts|cyberark).)*$

btool:

/opt/splunk/etc/system/default/outputs.conf        [syslog]
/opt/splunk/etc/system/default/outputs.conf        maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf        priority = <13>
/opt/splunk/etc/system/default/outputs.conf        type = udp
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf        ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf        autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf        autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf        blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf        blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf        cipherSuite = <removed>
/opt/splunk/etc/system/default/outputs.conf        compressed = false
/opt/splunk/etc/system/default/outputs.conf        connectionTimeout = 20
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf defaultGroup = rapidreader
/opt/splunk/etc/system/default/outputs.conf        disabled = false
/opt/splunk/etc/system/default/outputs.conf        dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf        dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf        ecdhCurves = <removed>
/opt/splunk/etc/system/default/outputs.conf        forceTimebasedAutoLB = false
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf forwardedindex.0.blacklist = ^((?!alerts|cyberark).)*$
/opt/splunk/etc/system/default/outputs.conf        forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf        heartbeatFrequency = 30
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf indexAndForward = true
/opt/splunk/etc/system/default/outputs.conf        maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf        maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf        maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf        readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf        secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf        sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf        sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf        sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf        tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf        useACK = false
/opt/splunk/etc/system/default/outputs.conf        writeTimeout = 300
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf [tcpout:rapidreader]
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf sendCookedData = false
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf server = IP:PORT

The regex blacklists everything except indexes containing "alerts" or "cyberark". You could make this even more robust by making that regex be an exact match not a containing match. Also just to detail my use case a bit more in case it helps anyone out - CyberArk data is generated from CyberArk and forwarding on to IDR, while the alerts index is populated using the "Log Event" alert action option in the Splunk alerts web interface. Both work and are forwarded to IDR!

I attempted to change forwardedindex.0.blacklist to a whitelist and that still sent everything. Also if the third party system goes down then it will likely overload your TCP processor and either interfere with all data collection, just Exchange data collection, or turn splunkd red. So that's an unfortunate risk.

For those of you who have versions of Splunk past 7.2.0 where a reboot wipes the system/default/outputs.conf... this is how I was told by a Splunk engineer (Tom) to deal with it (I have not needed to do so yet):

So, as long as the parameter is named the same in default -- it will be overwritten by the local parameters (due to precedence). You should be able to name the parameter and leave it blank (for the one whitelist) -- and this will survive an upgrade. This is a bit of an assumption -- but, whitelist and blacklist are normally capped at 10 (0-9), e.g. forwardedindex.0.whitelist - forwardedindex.9.whitelist.

View solution in original post

0 Karma

scoyle391
New Member

Hey all,
I write the InsightIDR documentation over at Rapid7 and definitely understand your complaints. I worked with one of the posters to update our documentation with as much detail as possible.

I invite you all to look it over here: https://insightidr.help.rapid7.com/docs/splunk

PLEASE tell me what else is missing, what else can be added, clarified, etc. I know this is a very confusing config and I want to make sure we get it figured out so no one else needs to struggle with it.

Thanks,
Steph

0 Karma

nick405060
Motivator

I spent about 5 minutes reading it over - it seems accurate to me, although I am slightly rusty a few months later and am not any official by any means. That being said, I did set up another export to IDR (so rn I am exporting Splunk Alerts, CyberArk data, and VPN logs - a couple gigs a day) using this method a week or two ago and everything has been working swimmingly. For example, with our VPN logs, they are routed ASA -> Splunk -> IDR which I think shows the (now) flexibility and integration potential of both SIEMS.

I apologize if I have not been super on top of this for ya - but feel free to email or Splunk Answers with any further urgent questions.

0 Karma

scoyle391
New Member

Thanks! I'm glad you were able to review it and that it's working better. Reach out to me with any necessary updates to this documentation!

0 Karma

nick405060
Motivator

Okay so this post is going to detail what actually got it working for me. I'll leave my earlier answer and comments up for various reasons. If you want to forward all your data to the third party system, then you can use transforms/outputs/props per the third-party-system Splunk documentation or per the route-and-filter-data Splunk documentation. Otherwise this answer is the only way that I know of to selectively forward data without interfering with Splunk data collection, at least with Splunk 7.2.0.

IDR:

Event Source: Rapid7 Raw Data
Display Name: Splunk
Collection Method: Log Aggregator
Log Aggregator: Splunk
Port: PORT
Protocol: TCP (not encrypted)

Directories:

/opt/splunk/etc/system/local - No outputs or props or transforms
/opt/splunk/etc/system/default - The three forwardedindex stanzas must be commented out in outputs.conf
/opt/splunk/etc/apps/rapid7_idr - The only file here is outputs.

outputs.conf:

[tcpout:rapidreader]
server = IP:PORT
sendCookedData = false

[tcpout]
defaultGroup = rapidreader
indexAndForward = true
forwardedindex.0.blacklist = ^((?!alerts|cyberark).)*$

btool:

/opt/splunk/etc/system/default/outputs.conf        [syslog]
/opt/splunk/etc/system/default/outputs.conf        maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf        priority = <13>
/opt/splunk/etc/system/default/outputs.conf        type = udp
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf        ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf        autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf        autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf        blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf        blockWarnThreshold = 100
/opt/splunk/etc/system/default/outputs.conf        cipherSuite = <removed>
/opt/splunk/etc/system/default/outputs.conf        compressed = false
/opt/splunk/etc/system/default/outputs.conf        connectionTimeout = 20
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf defaultGroup = rapidreader
/opt/splunk/etc/system/default/outputs.conf        disabled = false
/opt/splunk/etc/system/default/outputs.conf        dropClonedEventsOnQueueFull = 5
/opt/splunk/etc/system/default/outputs.conf        dropEventsOnQueueFull = -1
/opt/splunk/etc/system/default/outputs.conf        ecdhCurves = <removed>
/opt/splunk/etc/system/default/outputs.conf        forceTimebasedAutoLB = false
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf forwardedindex.0.blacklist = ^((?!alerts|cyberark).)*$
/opt/splunk/etc/system/default/outputs.conf        forwardedindex.filter.disable = false
/opt/splunk/etc/system/default/outputs.conf        heartbeatFrequency = 30
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf indexAndForward = true
/opt/splunk/etc/system/default/outputs.conf        maxConnectionsPerIndexer = 2
/opt/splunk/etc/system/default/outputs.conf        maxFailuresPerInterval = 2
/opt/splunk/etc/system/default/outputs.conf        maxQueueSize = auto
/opt/splunk/etc/system/default/outputs.conf        readTimeout = 300
/opt/splunk/etc/system/default/outputs.conf        secsInFailureInterval = 1
/opt/splunk/etc/system/default/outputs.conf        sendCookedData = true
/opt/splunk/etc/system/default/outputs.conf        sslQuietShutdown = false
/opt/splunk/etc/system/default/outputs.conf        sslVersions = tls1.2
/opt/splunk/etc/system/default/outputs.conf        tcpSendBufSz = 0
/opt/splunk/etc/system/default/outputs.conf        useACK = false
/opt/splunk/etc/system/default/outputs.conf        writeTimeout = 300
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf [tcpout:rapidreader]
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf sendCookedData = false
/opt/splunk/etc/apps/rapid7_idr/local/outputs.conf server = IP:PORT

The regex blacklists everything except indexes containing "alerts" or "cyberark". You could make this even more robust by making that regex be an exact match not a containing match. Also just to detail my use case a bit more in case it helps anyone out - CyberArk data is generated from CyberArk and forwarding on to IDR, while the alerts index is populated using the "Log Event" alert action option in the Splunk alerts web interface. Both work and are forwarded to IDR!

I attempted to change forwardedindex.0.blacklist to a whitelist and that still sent everything. Also if the third party system goes down then it will likely overload your TCP processor and either interfere with all data collection, just Exchange data collection, or turn splunkd red. So that's an unfortunate risk.

For those of you who have versions of Splunk past 7.2.0 where a reboot wipes the system/default/outputs.conf... this is how I was told by a Splunk engineer (Tom) to deal with it (I have not needed to do so yet):

So, as long as the parameter is named the same in default -- it will be overwritten by the local parameters (due to precedence). You should be able to name the parameter and leave it blank (for the one whitelist) -- and this will survive an upgrade. This is a bit of an assumption -- but, whitelist and blacklist are normally capped at 10 (0-9), e.g. forwardedindex.0.whitelist - forwardedindex.9.whitelist.
0 Karma

nick405060
Motivator

This may seem obvious but for a second output just add a [tcpout:rapidreader2] and second [tcpout] stanza, set defaultGroup in the second [tcpout] to rapidreader2, and then adjust your HOST/PORT/blacklist accordingly.

0 Karma

nick405060
Motivator

Hi there,

I have completed the Splunk-IDR integration and am forwarding/posting the following information to the R7 Documentation team, R7 support ticket, and Splunk support ticket that I have open. EDIT: This is not completely working. All of my Splunk data is forwarded to IDR and not a subset. In addition it is interfering with my Exchange collection.

Working configuration (directory is app local NOT system local):

'-- IDR -- '
I added Splunk with the product type "Rapid7" with TCP chosen as the protocol. In my case I opened up the port so that I could telnet both TCP and UDP; you should be fine with just TCP

'-- app.conf --'
[install]
is_configured = 1

'-- idrsetup.conf --'
[setupentity]
hostnames = (no port)

'-- indexes.conf --'
I don't remember if this file was empty or not upon app installation but I wiped it

'-- outputs.conf --'
[tcpout:rapidreader]
server = IP:PORT
sendCookedData = false

'-- props.conf --'
[index::cyberark]
TRANSFORMS-cyberark = rapid
[host::MYHOST]
TRANSFORMS-myhost = rapid
In this example I have two different items forwarded to IDR by both index and host

'-- transforms.conf --'
[rapid]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = rapidreader

Comments:
*Do NOT use R7's documentation here: https://insightidr.help.rapid7.com/docs/splunk
*Use https://docs.splunk.com/Documentation/Splunk/7.2.4/Forwarding/Forwarddatatothird-partysystemsd

R7 documentation flaws:
*Fails to mention that you NEED to add sendCookedData = false to outputs.conf since IDR only accepts raw data.
*States that you need "DEST_KEY = _SYSLOG_ROUTING" in your transforms.conf. This is completely false. You need _TCP_ROUTING for TCP, _SYSLOG_ROUTING for Syslog which their documentation does not cover. I didn't get this successfully set up with syslog, but if you want to use Syslog over TCP I strongly recommend to use the Splunk documentation which covers syslog.
*you should not be doing tihs in system local. Splunk documentation certainly does not tell you to do this in system local (although it does specify system local for syslog). Since it's a heirarchy and you can just use btool it doesn't matter a ton, but it is extremely inelegant programming... you should put the confs for an app inside the app.
*The documentation fails to mention how you do this for syslog
*The documentation fails to mention how you do this for UDP/if you can
*The default options in the R7 app interface make their changes in app local and not system local. But then the documentation tells you to do everything in system local. Smh.
*If you don't have the port open or have it open for the wrong protocol (in your firewall OR in R7), then your TCP processor gets overloaded and crashes your entire splunkd for your indexer. Exchange data ingestion specifically is affected/stopped. This should at the very least be cautioned somewhere in the documentation.
*Why are there not more default options in the R7 Splunk app interface
*Overall the documentation is too short, incomplete, and incorrect.
*others

linuxchuck
Explorer

Confirmed working here. I initially found and applied all the same changes except for switching it back to _TCP_ROUTING. I mistakenly used the R7 documentation, and set it to _SYSLOG_ROUTING, which caused it to fail. Once I corrected that, I now have events populating my Insight console through the relevant connectors. Thanks for the solid info!
Rapid7 would be better off just deleting their documentation, as it would reduce a ton of confusion around this integration.

0 Karma

nick405060
Motivator

Awesome. I still run into problems with it interfering with my Exchange data collection, but working on it

0 Karma

linuxchuck
Explorer

I agree on the documentation-fail at R7.

We are also trying to get the same integrations to work this week. If I come across any successes, I'll share them. I'd appreciate the same if you make any progress.

Thanks, and good luck!

0 Karma

scoyle391
New Member

I am the writer for the InsightIDR documentation and we just updated the Splunk docs. If you could take a look to see what else is missing / what needs to be clarified, we can update the doc asap!

0 Karma

nick405060
Motivator

Awesome! Same, testing out a bunch of different configurations this week, will definitely inform you of any luck that I have. the system vs app local thing in their documentation is wrong...

0 Karma

nick405060
Motivator

@linuxchuck Solved for me, see my below answer. it is the end of the day and i need to run, i know there were some formatting problems in my answer below and i will fix those tomorrow

0 Karma

linuxchuck
Explorer

Fully agree on all points below. I also refused to make changes in system/local, and instead applied them in the app local folder. I initially set them up per Splunk-specifications with _TCP_ROUTING, but changed them last-minute after making the mistake of reading R7 documentation. I've since returned them to the original config this morning, and will update you (+upvote the solution) once we confirm they are up and running.

0 Karma

nick405060
Motivator

@linuxchuck Were you able to send a subset of Splunk data or did it just send all of it? I am still having trouble specifying data to send. If you could post your confs that would amazing

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...