I have become intimately familiar with the eStreamer TA over the last couple of years. Let me see if I can help with some of these. setup.xml was removed in v4.0.x, so the configuration that was previously done with two passes through setup.xml in the GUI or TA-eStreamer/local/encore.conf now has to be done by manually editing the TA-eStreamer/bin/encore/estreamer.conf file, which is not nearly as easy-peasy as using the GUI. Packets, Connections, & Metadata (not mentioned earlier - but seems worth noting since it could be a data hog and is completely left out of the new instructions) In addition to manually enabling and setting the hosts in TA-eStreamer/bin/encore/estreamer.conf, you also have to manually enable/disable packets, connections, and metadata options that were previously available via checkboxes on the bottom of the setup page. packets are enabled by default - which could be problematic since packet data is quite large these options are in the "records" section of estreamer.conf as info, our previous configuration which had connections enabled, but packets and metadata disabled is below. "records": {
"connections": true,
"core": true,
"excl@comment": [
"These records will be excluded regardless of above (overrides 'include')",
"e.g. to exclude flow and IPS events use [ 71, 400 ]"
],
"exclude": [],
"inc@comment": "These records will be included regardless of above",
"include": [],
"intrusion": true,
"metadata": false,
"packets": false,
"rna": true,
"rua": true
}
}, Data Directory Change (affects inputs.conf & clean() function of splencore.sh script) As noted above the data directory has changed to TA-eStreamer/bin/encore/data/splunk/ The filename format has also changed from encore.EPOCHTIME.log to encore.logEPOCHTIME There are multiple ways you can address this. Either change where the data lives or point everything to the new locale. Change Where the Data Lives Update the "uri" in the "handler" section of TA-eStreamer/bin/encore/estreamer.conf back to the old value: "uri": "relfile:///../../data/encore.{0}.log" Update Where the App Looks Add a new monitor stanza in TA-eStreamer/local/inputs.conf for the new data path: [monitor://$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/data/splunk] Update the path in the clean() stanza of the TA-eStreamer/bin/splencore.sh script to the new data path: clean() {
# Delete data older than 12 hours -> 720mins
# find ../../data -type f -mmin +720 -delete
# correcting path to new path in new version 4.0.11 of TA
find $SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/data/splunk -type f -mmin +720 -delete
} first_pkt_sec EVAL Error The EVAL statement triggering the error looks like it was a FIELDALIAS that someone switched over to an EVAL without actually switching it. The culprit: EVAL-first_pkt_sec = event_sec as first_pkt_sec The fancy EVAL we wrote to address this coalesces several time fields to ensure the 'first_pkt_sec' field gets populated. EVAL-first_pkt_sec = coalesce(first_pkt_sec, connection_sec, event_sec) You could also accomplish this with a simple eval that will override the EVAL triggering the issue. EVAL-first_pkt_sec = event_sec Other Props Fixes We also noted that the search-time props had conflicting FIELDALIAS functions, no KV_MODE, and a few other things; so we added some additional flare to address those issues. Just in case this might also be helpful. [cisco:estreamer:data]
#### Setting the time format to epoch time (not set in TA)
TIME_FORMAT = %s
#### Setting KV_MODE ####
KV_MODE = auto
#### Splunk CIM - Intrusion Detection Fields ####
EVAL-severity = coalesce(severity, priority)
EVAL-signature = coalesce(case(signature="",null(),true(),signature), detection, msg)
#### Splunk CIM - Malware Fields ####
EVAL-url = coalesce(url, uri) If it wasn't clear - the first_pkt_sec and "other props fixes" were all applied to our TA-eStreamer/local/props.conf file.
... View more