how to forward Security Intelligence Event from eStreamer?

[haoban]

I'm using heavy forwarder and installed "Cisco eStreamer eNcore Add-on for Splunk" App to collect all the connection events from Cisco FMC.
Because of the Enterprise License limits, I only want to forward the "Security Intelligence Event" to the Indexer.
Now I can search all the events in Enterprise which forward from the forwarder.
I create props.conf and transforms.conf in the Heavy Forwarder under folder "/opt/splunk/etc/apps/TA-eStreamer/local", but seems it doesn't work.

props.conf
[cisco:estreamer:data]
TRANSFORMS-set = setnull

transforms.conf
[setnull]
REGEX = (sec_intel_event=Yes)
DEST_KEY = queue
FORMAT = nullQueue

Please help me to find out what's the issue. Thanks!

[hatalla]

@haoban - thanks for taking the time to provide all this. Much appreciated.

[haoban]

YES, the "Connection Events" is on the FMC side. TA-eStreamer is only used to receive the Events from the FMC and can do some filter. I don't remember the other configurations on the TA-eStream configuration page. I'll check my documents on Monday and give you more information.

Yes, I used "props.conf" and "transforms.conf" to only forward the "Intelligence Events" to the Splunk. You can follow the Splunk's documents to modify it by yourself, if you need some other filter conditions.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/propsconf

https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Transformsconf

[haoban]

@hatalla you need to choose the "Connection Events"
Login FMC, go to "System" -> "Integration" -> "eStreamer" -> "Connection Events"
But be careful, the data grows very fast. I'm using a Splunk forward as a filter, only forward the Intelligence Event to the Splunk Enterprise. And you also need to set the "Forward data" and "Receive data" in the Splunk forward and Splunk Enterprise

/opt/splunk/etc/apps/TA-eStreamer/local# cat props.conf
[cisco:estreamer:data]
TRANSFORMS-forward-data-to-receiver = forward_receiver

[cisco:estreamer:log]
TRANSFORMS-drop-data = setdrop

[cisco:estreamer:status]
TRANSFORMS-drop-data = setdrop

/opt/splunk/etc/apps/TA-eStreamer/local# cat transforms.conf
[forward_receiver]
REGEX = sec_intel_list1
DEST_KEY = _TCP_ROUTING
FORMAT = default-autolb-group

[setdrop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[hatalla]

@haoban - Thanks for your response. Ok I'll work with the Firepower admin to configure the FMC to send the connection events; so it seems it is a setting on the FMC side and not on the TA-eStreamer side; is that correct? Reason I am seeing is that on the TA-eStreamer add-configuration page there a setting under "data" to collect the "connections" though it doesn't seem it is doing anything when checked.

As far as transforms.conf - it seems to me you are trying to ingest in Splunk ONLY the security intelligence traffic, hence the REGEX = sec_intel_list1 then sending that stream to your tcpout group name(s) in outputs.conf while your tcpout stanza in outputs.conf is called "default-autolb-group" and sending everything else (hence REGEX = .) to the nullQueue - is that what you are trying to do?

Thanks.

[haoban]

Please notice I used 2 Servers here, one is Splunk forward , another is Splunk enterprise.
"Cisco eStreamer eNcore Add-on for Splunk" install on Splunk forward.
"Cisco Firepower eNcore App for Splunk" install on Splunk enterprise.
"Geo Location Lookup Script (powered by MAXMIND)" install on Splunk enterprise.
GeoLite2 databases unzip on Splunk enterprise.

Installation

Download "Cisco eStreamer eNcore Add-on for Splunk" from https://splunkbase.splunk.com/app/3662/ (cisco-estreamer-encore-add-on-for-splunk_356.tgz)
Download "Cisco Firepower eNcore App for Splunk" from https://splunkbase.splunk.com/app/3663/ (cisco-firepower-encore-app-for-splunk_353.tgz)
Download "Geo Location Lookup Script (powered by MAXMIND)" from https://splunkbase.splunk.com/app/291/
Download GeoLite2 databases from
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz

down XXXX.pkcs12 from FMC and upload it on Splunk forward here: $SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/client.pkcs12
PKCS12 file must be renamed as “client.pkcs12”

Configuration on Splunk forward

Edit "/opt/splunk/etc/apps/TA-eStreamer/bin/configure.sh"
Modify "exec &>configuration.log" to "exec >>configuration.log 2>&1"

Navigate to app settings in Splunk – from the home page, click the “cog” icon
Find Cisco eStreamer eNcore for Splunk and click “Set-up”
At a minimum:

■ enter the “FMC hostname or IP address” and
IP XXX.XXX.XXX.XXX
Port 8302

■ check the “Process PKCS12 file?”. No password here

Note: Each time you load this page, “Process PKCS12 file” is reset to “no” and the password is not saved. It should be used once to process the PKCS12 file using openSSL and store a public-private key pair.

Select "Packets? Packet logs can be large and use up storage" and "Connections? This is a very high-volume option and may consume significant network and storage usage"
Click "Save"

Enable the data inputs on Splunk forward

Navigate to Settings > Data Inputs > Files & Directories and enable the single TA-eStreamer app input (cisco:estreamer:data) – this is the where the main output data files are saved

Navigate to Settings > Data Inputs > Scripts and enable the three TA-eStreamer inputs

■ cisco:estreamer:clean – this script has no output but is used to delete data files older than 12 hours

■ cisco:estreamer:log – this script uses the stdout of eNcore to take program log data. This becomes very useful where things are not going to plan

■ cisco:estreamer:status – this script runs periodically to maintain a clear status of whether the program is running or not

Execution on Splunk forward

Once you have fully configured the collector and enabled the inputs, navigate back to the set-up page in app settings, enable eNcore (“is enabled?”) and press save.
To check the status, search for sourcetype="cisco:estreamer:status"
To check more detailed log output, search for sourcetype="cisco:estreamer:log"
To look for eStreamer data, search for sourcetype=" cisco:estreamer:data"

Troubleshooting

If you see Error while posting to url=/servicesNS/nobody/TA-eStreamer/encore/configure/main when you press Save in the setup screen, then please search the logs for more information: Search: index=_internal source="*splunkd.log" AdminManagerExternal

If you are getting less data than you are expecting or just want to see what the eStreamer client is doing, then search: sourcetype="cisco:estreamer:log" (ERROR OR WARNING). To see more detail, remove the ERROR and WARNING constraints.

replace GeoLite2-City.mmdb on Splunk Enterprise

upload "GeoLite2-City_yyyymmdd.tar.gz" to the Splunk server path /opt/splunk/share
tar -xvf GeoLite2-City_yyyymmdd.tar.gz
cd GeoLite2-City_yyyymmdd
cp GeoLite2-City.mmdb ../

filter the logs on Splunk forward

cd /opt/splunk/etc/apps/TA-eStreamer/local

==props.conf==
[cisco:estreamer:data]
TRANSFORMS-forward-data-to-receiver = forward_receiver

[cisco:estreamer:log]
TRANSFORMS-drop-data = setdrop

[cisco:estreamer:status]
TRANSFORMS-drop-data = setdrop

==transforms.conf==
[forward_receiver]
REGEX = sec_intel_list1
DEST_KEY = _TCP_ROUTING
FORMAT = default-autolb-group

[setdrop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

configure forward on Splunk forward

settings->Forwarding and receiving->Configure forwarding "Add new"
Host: xxx.xxx.xxxx.xxx:9997

configure receiving on Splunk Enterprise

settings->Forwarding and receiving->Configure receiving "Add new"
Listen on this port:9997

[hatalla]

Hey Haoban - how are you getting the security intelligence logs from Firepower? I am also using the same TA (TA-eStreamer) but the only traffic that seems to be coming from Firepower is intrusion detection and malware events. I am not seeing any events with sec_intel_event=yes

[haoban · Accepted solution]

/opt/splunk/etc/apps/TA-eStreamer/local# cat props.conf
[cisco:estreamer:data]
TRANSFORMS-send-data-to-null-queue = setnull

/opt/splunk/etc/apps/TA-eStreamer/local# cat transforms.conf
[setnull]
REGEX = (sec_intel_event=No)
DEST_KEY = queue
FORMAT = nullQueue