I installed the FireEye application and configured my WebMPS sensors to post alerts to Splunk per the instructions. I've reviewed the Splunkd log and can see where the events are making their way to Splunk.
The default FireEye dashboards have no results. I know the events are in the index because if I search for index=fe, my events show up.
Inspecting one of the dashboards shows the following query being run with no results:
search index=fe sourcetype="fe_xml" fe_appliance=* src_ip=* malware_name=* alert_id=* | rename severity AS fe_severity | eval fe_severity=replace(fe_severity,"\"","") | lookup severity fe_severity | rename severity AS SEVERITY | table alert_id fe_appliance src_ip dest_ip malware_name malware_stype SEVERITY
If I run this query manually, I get no results. If I start backing parameters of the query off, I get results at
index=fe sourcetype="fe_xml" fe_appliance=*
As soon as src_ip=* is tacked onto the search, results stop coming in.
I haven't modified any of the FireEye application files, and I've triple-checked that my sensors are sending data to Splunk in the Extended XML format. Does anyone have any ideas why this isn't working correctly? Thanks!
EDIT:
Looks like the src_ip isn't being extracted. I wonder if the event format changed? Here's an edited raw event:
<?xml version="1.0" encoding="utf-8"?><alerts appliance="ApplianceName" msg="extended" product="Web MPS" version="7.0.2.156588" xmlns="hxxp://www.fireeye.com/alert/2011/AlertSchema" xmlns:xsi="hxxp://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="hxxp://www.fireeye.com/alert/2011/AlertSchema FireEyeAlert.xsd"><alert id="405294" name="infection-match" severity="minr"><explanation analysis="content" protocol="tcp"><malware-detected><malware name="Local.Infection" sid="600104" stype="bot-command"/></malware-detected><cnc-services><cnc-service port="80" protocol="tcp"><address>141.101.117.X</address><channel>GET /efax_7132159010.doc hxxp/1.1::~~Accept: */*::~~User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; ms-office; MSOffice 14)::~~Accept-Encoding: gzip, deflate::~~Host: slash.ma::~~Connection: Keep-Alive::~~Cookie: __cfduid=db3535ebf75b3f2976167c05678ac6e631392738961234::~~::~~</channel></cnc-service></cnc-services></explanation><src vlan="0"><ip>192.168.12.150</ip><host>host.domain.local</host><port>58268</port><mac>00:15:c7:00:00:00</mac></src><dst><ip>141.101.117.x</ip><mac>00:1b:54:f7:00:00</mac><port>80</port></dst><occurred>2014-02-18T18:00:16Z</occurred><interface label="A1" mode="tap">pether3</interface><alert-url>hxxps://sensor/event_stream/events_for_bot?ev_id=405294&lms_iden=00:E0:81:C3:00:00</alert-url><action>notified</action></alert></alerts>
A Universal Solution (Works for any version of FireEye OS) should be below:
FireEye 7.1 and later uses "Pretty print" for the XML. While this increases the readability for humans, it does break the Splunk parsing in the current app. We can account for the extra carriage returns, line feeds, and tabs using the following: ([\r\n]*)\s*
There appear to be two Extraction parameters that need a slight modification to make all of the dashboards function again: EXTRACT-src-dst and EXTRACT-malwaredetected
From the Splunk GUI:
App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"
Click on the top hyperlink and replace the current contents with the following:
<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<host>(?<src_host>.*)</host>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>
Notice that the old regex did not take into account the <host>
tag which also broke the parsing. This new one does.
Now change EXTRACT-malwaredetected...
From the Splunk GUI:
App -> Manage Apps -> View Objects -> Search box: "EXTRACT-malwaredetected"
Click on the top hyperlink and replace the current contents with the following:
<malware-detected>([\r\n]*)\s*<malware\sname=\"(?<malware_name>.*)\"\s+sid=\"(?<malware_sid>.*)\"\s+stype=\"(?<malware_stype>.*)\"/> in _raw
Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd
Let me know if this works for you. Thanks.
Thanks for hanging in there. Please try out our new app designed to work with Splunk v6. We increased the number of FireEye appliances we support and drastically increased the number of protocols you can use to send data to Splunk. Download it here: http://apps.splunk.com/app/1845/. It is still in BETA, but we are accepting user feedback at: Tony.Lee -at- FireEye.com. Thanks!
A Universal Solution (Works for any version of FireEye OS) should be below:
FireEye 7.1 and later uses "Pretty print" for the XML. While this increases the readability for humans, it does break the Splunk parsing in the current app. We can account for the extra carriage returns, line feeds, and tabs using the following: ([\r\n]*)\s*
There appear to be two Extraction parameters that need a slight modification to make all of the dashboards function again: EXTRACT-src-dst and EXTRACT-malwaredetected
From the Splunk GUI:
App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"
Click on the top hyperlink and replace the current contents with the following:
<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<host>(?<src_host>.*)</host>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>
Notice that the old regex did not take into account the <host>
tag which also broke the parsing. This new one does.
Now change EXTRACT-malwaredetected...
From the Splunk GUI:
App -> Manage Apps -> View Objects -> Search box: "EXTRACT-malwaredetected"
Click on the top hyperlink and replace the current contents with the following:
<malware-detected>([\r\n]*)\s*<malware\sname=\"(?<malware_name>.*)\"\s+sid=\"(?<malware_sid>.*)\"\s+stype=\"(?<malware_stype>.*)\"/> in _raw
Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd
Let me know if this works for you. Thanks.
Side note, some of the events may or may not have a "host" field in them, which provides some inconsistencies with how the src-dst extractions work. My workaround just put another field extraction in place to catch the ones that didn't have a host field:
<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>
So others don't need to to make this change... We patched the Splunk for FireEye app to version 2.0.8 (making the change above) and pushed it to the Splunk apps site. I also created a howto for installation on Splunk v6 as a stopgap until the new app is completed. See the instructions below:
http://securitysynapse.com/2014/05/stopgap-splunk-for-fireeye-v2-app.html
Let us know if you have any issues in the mean time,
-Tony
Try using the Universal fix before this one. I believe that should solve the problem no matter the version of FE OS. This will be left here just in case the other solution does not work for you.
Here is the fix for FireEye (pre-OS 7.1) appliances. This should work if your wMPS (NX) appliances are any version prior to 7.1. I will post a universal fix (that works for all appliances) if possible.
From the Splunk GUI:
App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"
Click on the top hyperlink and replace the current contents with the following:
<src\svlan=\"\d+\"\><ip>(?<src_ip>[0-9.]+)</ip><host>(?<src_host>.*)</host><port>(?<src_port>\d+)</port><mac>(?<src_mac>[\w:]+)</mac></src><dst><ip>(?<dest_ip>[0-9.]+)</ip><mac>(?<dest_mac>[\w:]+)</mac><port>(?<dest_port>\d+)</port></dst>
Alternatively, you can also change this directly in the props.conf file:
Original:
EXTRACT-src-dst = \<src\svlan=\"\d+\"\>\<ip\>(?<src_ip>[0-9.]+)\</ip\>\<port\>(?<src_port>\d+)\</port\>\<mac\>(?<src_mac>[\w:]+)\</mac\>\</src\>\<dst\>\<ip\>(?<dest_ip>[0-9.]+)\</ip\>\<mac\>(?<dest_mac>[\w:]+)\</mac\>\<port\>(?<dest_port>\d+)\</port\>\</dst\>
Change to:
EXTRACT-src-dst = <src\svlan=\"\d+\"\><ip>(?<src_ip>[0-9.]+)</ip><host>(?<src_host>.*)</host><port>(?<src_port>\d+)</port><mac>(?<src_mac>[\w:]+)</mac></src><dst><ip>(?<dest_ip>[0-9.]+)</ip><mac>(?<dest_mac>[\w:]+)</mac><port>(?<dest_port>\d+)</port></dst>
Notice that the old regex did not take into account the
Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd
No problem Dave. Let me know if the Universal fix works for you. I tested it on FireEye OS 6.2 and 7.1 and it seems to be working.
Thanks so much Tony, I am running 7.1.1 so if you get a chance for the universal fix, that would be great. In the mean time, I am going to see if the above works or can be manipulated to work for me.
I am having the same issue with src_ip not being extracted. Did you ever figure this one out?
I'd check to make sure the fe_appliance, src_ip, malware_name and alert_id fields are being extracted. I'd do a search for index=fe sourcetype="fe_xml" and see if the field shows up on the left hand side.
Brian
I am trying to extract the src_ip and malware_name fields and when I have Splunk write the regex in the "Interactive Field Extractor", it can't find the src_ip line probably because it is only pulling in the first 15 lines. Can I get this expanded? I could upload a screen shot if this is not clear...
Keep in mind that there are different "names" within the events. For example:
One "name" contained within the
Another "name" contained within the
The root cause of the field extraction issue may be located in the props.conf or the transforms.conf file?
Same with me, src_ip and malware_name (though name is) are not extracted. fe_appliance and alert_id are also fine
Thanks Brian. It appears that src_ip and malware_name are not being extracted properly. fe_appliance and alert_id are fine.
I couldn't post an event other than by editing the original post, but I now have one of the events in my post. I'm wondering if the format changed?