Latest eStreamer not CIM compliant?

[bmorgenthaler]

I've installed the latest TA-eStreamer and I'm trying to see if I can get the data into InfoSec App for Splunk for IDS/IDP events. I followed the setup instructions and I can see data coming in. Unfortunately the events do not appear to be tag and aren't getting put into the CIM Data Model for Network Traffic or Intrusion Detection. Looking at the props.conf there are a number of entries for CIM fields but I'm not getting them recognized.

[bmorgenthaler · Accepted solution]

I THINK I solved my problem. It appears that the latest eNcore eStreamer does not have eventtypes tagged. So I added the following two files, which I jammed together from the previous eStreamer version, to my local configs.

TA-eStreamer/local/tags.conf

[eventtype=estreamer-ids-ips-event]
ids = enabled
attack = enabled

[eventtype=estreamer-file-malware-event]
malware = enabled
attack = enabled

[eventtype=estreamer-scan-event]
malware = enabled
operations = enabled

[eventtype=estreamer-flow]
network = enabled
communicate = enabled

[eventtype=estreamer-si-event]
network = enabled
communicate = enabled

TA-eStreamer/local/eventtypes.conf

[estreamer-ids-ips-event]
search = sourcetype=cisco:estreamer:data rec_type_simple="IPS EVENT"

[estreamer-file-malware-event]
search = sourcetype=cisco:estreamer:data (rec_type_simple="FILELOG EVENT" OR rec_type_simple="FILELOG MALWARE EVENT" OR rec_type_simple="MALWARE EVENT") NOT sha256=""

[estreamer-scan-event]
search = sourcetype=cisco:estreamer:data (rec_type_simple="FILELOG EVENT" OR rec_type_simple="FILELOG MALWARE EVENT" OR rec_type_simple="MALWARE EVENT") sha256=""

[estreamer-flow]
search = sourcetype=cisco:estreamer:data rec_type_simple=RNA event_type=1003

[estreamer-si-event]
search = sourcetype=cisco:estreamer:data rec_type_simple=RNA event_type=1003 sec_intel_event=Yes

As of right now this is working for me.

[adobrzeniecki]

I am seeing in the eStreamer-Dashboard app that those eventtypes exist. Is this correct? Should they be in the TA or App?

[igifrin_splunk]

Hi @bmorgenthaler, just to confirm that you are correct - the eStreamer add-on (version 3.6.1 at the time of this message) does not have tags even though it is marked by the developer as CIM-compliant.

Thanks for sharing the tags.conf and eventtypes.conf files.

[dagar_ruralking]

@bmorgenthaler, I know this thread is 6 months old. I had emailed fp-4-splunkATcisco.com regarding this. I am trying to work with the FirePower devs to get this resolved. I created an event type and tag for the malware event like you had above, but I get a lot more events than what is malware. Have you had any luck in cleaning this up?

[marycordova]

please post your inputs and your effective props configs

[bmorgenthaler]

Here is my effective props for the cisco:estreamer sourcetype.

$ ./splunk cmd btool props list cisco:estreamer
[cisco:estreamer:data]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
FIELDALIAS-estreamer_app = app_proto AS app
FIELDALIAS-estreamer_bytes_in = src_bytes AS bytes_in
FIELDALIAS-estreamer_bytes_out = dest_bytes AS bytes_out
FIELDALIAS-estreamer_category = malware_event_type AS malware_type
FIELDALIAS-estreamer_connection_id = connection_counter AS connection_id
FIELDALIAS-estreamer_date = event_sec AS date
FIELDALIAS-estreamer_dest = dest_ip AS dest
FIELDALIAS-estreamer_dest_interface = iface_egress AS dest_interface
FIELDALIAS-estreamer_dest_zone = sec_zone_egress AS dest_zone
FIELDALIAS-estreamer_dvc = sensor AS dvc
FIELDALIAS-estreamer_file_hash = sha256 AS file_hash
FIELDALIAS-estreamer_first_pkt_sec_1 = connection_second AS first_pkt_sec
FIELDALIAS-estreamer_first_pkt_sec_2 = connection_sec AS first_pkt_sec
FIELDALIAS-estreamer_flow_id = connection_id AS flow_id
FIELDALIAS-estreamer_instance_id = connection_instance_id AS instance_id
FIELDALIAS-estreamer_intrusion_signature = msg AS signature
FIELDALIAS-estreamer_malware_signature = detection AS signature
FIELDALIAS-estreamer_packets_in = src_pkts AS packets_in
FIELDALIAS-estreamer_packets_out = dest_pkts AS packets_out
FIELDALIAS-estreamer_rule = fw_rule AS rule
FIELDALIAS-estreamer_severity = priority AS severity
FIELDALIAS-estreamer_src = src_ip AS src
FIELDALIAS-estreamer_src_interface = iface_ingress AS src_interface
FIELDALIAS-estreamer_src_zone = sec_zone_ingress AS src_zone
FIELDALIAS-estreamer_tcp_flag = tcp_flags AS tcp_flag
FIELDALIAS-estreamer_url = uri AS url
FIELDALIAS-estreamer_vlan = vlan_id AS vlan
HEADER_MODE =
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
LOOKUP-estreamer_file_action = file_actions file_action OUTPUT action
LOOKUP-estreamer_fw_action = fw_actions fw_rule_action OUTPUT action
LOOKUP-estreamer_severities = severities impact,priority OUTPUT severity
LOOKUP-estreamer_sources = sources source OUTPUT vendor, product, ids_type
LOOKUP-estreamer_transport = ip_protos ip_proto OUTPUT transport
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = false
TIME_PREFIX = event_sec=
TRANSFORMS =
TRUNCATE = 0
detect_trailing_nulls = false
maxDist = 100
priority =
sourcetype =
[cisco:estreamer:log]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
EXTRACT-encore_log_fields = ^(?P<timestamp>\d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+)\s+(?P<class>[^ ]+)\s+(?P<severity>\w+)\s+(?P<message>.+)
HEADER_MODE =
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
detect_trailing_nulls = false
maxDist = 100
priority =
sourcetype =

And here is my inputs for the app (configured as per the directions):

[monitor:///opt/splunk/etc/apps/TA-eStreamer/data]
_rcvbuf = 1572864
crcSalt = <SOURCE>
disabled = false
host = firesight
index = estreamer
source = encore
sourcetype = cisco:estreamer:data
[script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh clean]
_rcvbuf = 1572864
disabled = 0
host = bbmosplunk
index = default
interval = 900
source = encore
sourcetype = cisco:estreamer:clean
start_by_shell = false
[script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh start]
_rcvbuf = 1572864
disabled = 0
host = bbmosplunk
index = default
interval = 120
source = encore
sourcetype = cisco:estreamer:log
start_by_shell = false
[script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh status]
_rcvbuf = 1572864
disabled = 0
host = bbmosplunk
index = default
interval = 30
source = encore
sourcetype = cisco:estreamer:status
start_by_shell = false

[kchugh]

@bmorgenthaler I know this is an old thread but could also please share the contents of the lookup files?

LOOKUP-estreamer_file_action = file_actions file_action OUTPUT action
LOOKUP-estreamer_fw_action = fw_actions fw_rule_action OUTPUT action
LOOKUP-estreamer_severities = severities impact,priority OUTPUT severity
LOOKUP-estreamer_sources = sources source OUTPUT vendor, product, ids_type
LOOKUP-estreamer_transport = ip_protos ip_proto OUTPUT transport