topic Re: Cisco eStreamer eNcore - filtering not working in Getting Data In

Cisco eStreamer eNcore - filtering not working

halfreeman — Tue, 29 Sep 2020 22:54:17 GMT

Good day,

I'm trying to filter connection events from FMC eStreamer, i.e. I do not need Allowed connections in Splunk. I thought I had lurked enough into Splunk documentation and Community, but I guess - not. 😞 So I need help from senior admins.

I had created in /opt/splunk/etc/apps/TA-eStreamer/local
props.conf
[cisco:estreamer:data]
rename = fmc
TRANSFORMS-send-data-to-null-queue = setnull

transforms.conf
[setnull]
REGEX = (fw_rule_action=Allow)
DEST_KEY = queue
FORMAT = nullQueue

Restarted Splunk, and yet I still get
rec_type=71 file_count=0 client_app="SSL client" mac_address=00:00:00:00:00:00 dest_ip=... dest_port=443 sec_intel_ip=N/A
...
ssl_expected_action=Unknown app_proto=HTTPS ssl_server_name="" ssl_cert_fingerprint=0000000000000000000000000000000000000000 has_ipv6=1 fw_rule_action=Allow ssl_rule_id=0
...

I'm using latest eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Tue, 29 Sep 2020 22:54:23 GMT

HI,
you can discard whole rec_type like this:

transforms

[discard_rec_type_71]
REGEX=^rec_type=71
DEST_KEY=queue
FORMAT=nullQueue

Props.conf

[cisco:estreamer:data]
TRANSFORMS-rec_type71=discard_rec_type_71

Or do you try to discard fields within the rec_type=71 events? This is possible as well.

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Wed, 23 Jan 2019 07:38:46 GMT

Did you try this?

Please accept the answer if it helped

Re: Cisco eStreamer eNcore - filtering not working

halfreeman — Tue, 29 Sep 2020 22:54:52 GMT

Alas rec_type=71 are for all kinds of messages - Allow, Trust, Block. Block with reset.

I.e.
rec_type=71 file_count=0 client_app="SSL client" mac_address=00:00:00:00:00:00 dest_ip=...
...
ssl_server_name="" ssl_cert_fingerprint=0000000000000000000000000000000000000000 has_ipv6=1 fw_rule_action="Block with reset" ssl_rule_id=0 netbios_domain="" fw_rule="Bacis access - Deny Traffic"
...

Actually I either need to discard messages with fw_rule_action= equals Allow or Trust or keep only
Block and *Block with reset"

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Tue, 29 Sep 2020 22:56:10 GMT

You could try :

[discard_rec_type_71]
REGEX=fw_rule_action=("Allow"|"Trust")
DEST_KEY=queue
FORMAT=nullQueue

Re: Cisco eStreamer eNcore - filtering not working

halfreeman — Tue, 29 Sep 2020 22:54:55 GMT

No success.

Btw, just in case I tried "REGEX=^rec_type=71" and it worked - all such messages were filter out. So in general the idea is correct.

The question is in specific REGEX?

Here is whole typical "Allow" message:
rec_type=71 web_app=Weborama dest_port=443 file_count=0 dest_autonomous_system=0 sec_intel_ip=N/A netflow_src=00000000-0000-0000-0000-000000000000 http_response=0 dest_bytes=992 user_agent="" dns_resp_id=0 event_type=1003 dns_rec_id=0 client_app="SSL client" ssl_flow_error=0 legacy_ip_address=0.0.0.0 event_usec=0 vlan_id=0 ssl_actual_action=Unknown dns_query="" dest_ip_country=france snmp_out=0 ssl_flow_status=Unknown event_desc="Flow Statistics" src_mask=0 num_ioc=0 rec_type_desc="Connection Statistics" sensor=Onego event_subtype=1 security_context=00000000000000000000000000000000 sinkhole_uuid=00000000-0000-0000-0000-000000000000 fw_rule_action=Allow dest_pkts=4 fw_policy=00000000-0000-0000-0000-00005c47a1f6 url=https://adv.solution.weborama.fr tcp_flags=0 mac_address=00:00:00:00:00:00 has_ipv6=1 ssl_server_cert_status="Not Checked" fw_rule_reason=N/A ssl_version=Unknown dns_ttl=0 ssl_rule_id=0 app_proto=HTTPS ssl_flow_messages=0 ssl_cert_fingerprint=0000000000000000000000000000000000000000 iface_ingress=inside src_tos=0 referenced_host="" iface_egress=outside monitor_rule_1=N/A monitor_rule_3=N/A monitor_rule_2=N/A monitor_rule_5=N/A monitor_rule_4=N/A monitor_rule_7=N/A monitor_rule_6=N/A monitor_rule_8=0 instance_id=1 dest_ip=91.216.195.18 ip_layer=0 last_pkt_sec=1548241758 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 src_pkts=6 sec_intel_event=No dest_mask=0 snmp_in=0 rec_type_simple=RNA first_pkt_sec=1548241758 fw_rule="Extended access" ssl_policy_id=00000000000000000000000000000000 url_category=Uncategorized url_reputation="Risk unknown" ssl_url_category=0 ip_proto=TCP ssl_flow_flags=0 src_bytes=2022 event_sec=1548241938 sec_zone_ingress=Sanctum ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL netbios_domain="" src_autonomous_system=0 client_version="" src_ip_country=unknown ssl_ticket_id=0000000000000000000000000000000000000000 dest_tos=0 src_port=52189 http_referrer="" sec_zone_egress=Exodus src_ip=192.168.111.111 user=blahblahuser ips_count=0 ssl_server_name="" ssl_expected_action=Unknown connection_id=28348

Re: Cisco eStreamer eNcore - filtering not working

halfreeman — Tue, 29 Sep 2020 22:54:58 GMT

Actually - this one worked just need to remove quote marks REGEX=fw_rule_action=(Allow|Trust).

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Wed, 23 Jan 2019 11:43:01 GMT

Great pls accept the answer 🙂

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Tue, 29 Sep 2020 22:56:16 GMT

Small update, if you are interessed:

you can add this to make sure its only valid for rec_type 71

^rec_type=71.*?fw_rule_action=(Allow|Trust)

Re: Cisco eStreamer eNcore - filtering not working

rajashaey — Tue, 26 Sep 2023 10:53:11 GMT

hello, am looking for reference of field "rec_type" and what is actually means?
I tried searching cisco documentation but no luck. Please share with me? the link @halfreeman @dkeck

Re: Cisco eStreamer eNcore - filtering not working

dkeck — Wed, 27 Sep 2023 08:37:19 GMT

https://www.cisco.com/c/en/us/td/docs/security/firesight/540/api/estreamer/EventStreamerIntegrationGuide/IS-DCRecords.html

maybe this helps?

I am not really familiar with estreamer in details. What I understood from a splunk perspective is that, rec_type is the main identifier for the firewall events. The TA also use it to break the events. In the python script you are able to filter out rec_types and fields, based on rec_types.

David