All Apps and Add-ons

FireEye Built in Dashboards Not Working

wbkendall
Explorer

I installed the FireEye application and configured my WebMPS sensors to post alerts to Splunk per the instructions. I've reviewed the Splunkd log and can see where the events are making their way to Splunk.
The default FireEye dashboards have no results. I know the events are in the index because if I search for index=fe, my events show up.

Inspecting one of the dashboards shows the following query being run with no results:

search index=fe sourcetype="fe_xml" fe_appliance=* src_ip=* malware_name=* alert_id=* | rename severity AS fe_severity | eval fe_severity=replace(fe_severity,"\"","") | lookup severity fe_severity | rename severity AS SEVERITY | table alert_id fe_appliance src_ip dest_ip malware_name malware_stype SEVERITY

If I run this query manually, I get no results. If I start backing parameters of the query off, I get results at

index=fe sourcetype="fe_xml" fe_appliance=*

As soon as src_ip=* is tacked onto the search, results stop coming in.

I haven't modified any of the FireEye application files, and I've triple-checked that my sensors are sending data to Splunk in the Extended XML format. Does anyone have any ideas why this isn't working correctly? Thanks!

EDIT:

Looks like the src_ip isn't being extracted. I wonder if the event format changed? Here's an edited raw event:

<?xml version="1.0" encoding="utf-8"?><alerts appliance="ApplianceName" msg="extended" product="Web MPS" version="7.0.2.156588" xmlns="hxxp://www.fireeye.com/alert/2011/AlertSchema" xmlns:xsi="hxxp://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="hxxp://www.fireeye.com/alert/2011/AlertSchema FireEyeAlert.xsd"><alert id="405294" name="infection-match" severity="minr"><explanation analysis="content" protocol="tcp"><malware-detected><malware name="Local.Infection" sid="600104" stype="bot-command"/></malware-detected><cnc-services><cnc-service port="80" protocol="tcp"><address>141.101.117.X</address><channel>GET /efax_7132159010.doc hxxp/1.1::~~Accept: */*::~~User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; ms-office; MSOffice 14)::~~Accept-Encoding: gzip, deflate::~~Host: slash.ma::~~Connection: Keep-Alive::~~Cookie: __cfduid=db3535ebf75b3f2976167c05678ac6e631392738961234::~~::~~</channel></cnc-service></cnc-services></explanation><src vlan="0"><ip>192.168.12.150</ip><host>host.domain.local</host><port>58268</port><mac>00:15:c7:00:00:00</mac></src><dst><ip>141.101.117.x</ip><mac>00:1b:54:f7:00:00</mac><port>80</port></dst><occurred>2014-02-18T18:00:16Z</occurred><interface label="A1" mode="tap">pether3</interface><alert-url>hxxps://sensor/event_stream/events_for_bot?ev_id=405294&amp;lms_iden=00:E0:81:C3:00:00</alert-url><action>notified</action></alert></alerts>
Tags (1)
1 Solution

TonyLeeVT
Builder

A Universal Solution (Works for any version of FireEye OS) should be below:

FireEye 7.1 and later uses "Pretty print" for the XML. While this increases the readability for humans, it does break the Splunk parsing in the current app. We can account for the extra carriage returns, line feeds, and tabs using the following: ([\r\n]*)\s*

There appear to be two Extraction parameters that need a slight modification to make all of the dashboards function again: EXTRACT-src-dst and EXTRACT-malwaredetected

From the Splunk GUI:

App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"

Click on the top hyperlink and replace the current contents with the following:

<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<host>(?<src_host>.*)</host>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>

Notice that the old regex did not take into account the <host> tag which also broke the parsing. This new one does.


Now change EXTRACT-malwaredetected...

From the Splunk GUI:

App -> Manage Apps -> View Objects -> Search box: "EXTRACT-malwaredetected"

Click on the top hyperlink and replace the current contents with the following:

<malware-detected>([\r\n]*)\s*<malware\sname=\"(?<malware_name>.*)\"\s+sid=\"(?<malware_sid>.*)\"\s+stype=\"(?<malware_stype>.*)\"/> in _raw

Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd

Let me know if this works for you. Thanks.

View solution in original post

TonyLeeVT
Builder

Thanks for hanging in there. Please try out our new app designed to work with Splunk v6. We increased the number of FireEye appliances we support and drastically increased the number of protocols you can use to send data to Splunk. Download it here: http://apps.splunk.com/app/1845/. It is still in BETA, but we are accepting user feedback at: Tony.Lee -at- FireEye.com. Thanks!

0 Karma

TonyLeeVT
Builder

A Universal Solution (Works for any version of FireEye OS) should be below:

FireEye 7.1 and later uses "Pretty print" for the XML. While this increases the readability for humans, it does break the Splunk parsing in the current app. We can account for the extra carriage returns, line feeds, and tabs using the following: ([\r\n]*)\s*

There appear to be two Extraction parameters that need a slight modification to make all of the dashboards function again: EXTRACT-src-dst and EXTRACT-malwaredetected

From the Splunk GUI:

App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"

Click on the top hyperlink and replace the current contents with the following:

<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<host>(?<src_host>.*)</host>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>

Notice that the old regex did not take into account the <host> tag which also broke the parsing. This new one does.


Now change EXTRACT-malwaredetected...

From the Splunk GUI:

App -> Manage Apps -> View Objects -> Search box: "EXTRACT-malwaredetected"

Click on the top hyperlink and replace the current contents with the following:

<malware-detected>([\r\n]*)\s*<malware\sname=\"(?<malware_name>.*)\"\s+sid=\"(?<malware_sid>.*)\"\s+stype=\"(?<malware_stype>.*)\"/> in _raw

Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd

Let me know if this works for you. Thanks.

srioux
Communicator

Side note, some of the events may or may not have a "host" field in them, which provides some inconsistencies with how the src-dst extractions work. My workaround just put another field extraction in place to catch the ones that didn't have a host field:

<src\svlan=\"\d+\"\>([\r\n]*)\s*<ip>(?<src_ip>[0-9.]+)</ip>([\r\n]*)\s*<port>(?<src_port>\d+)</port>([\r\n]*)\s*<mac>(?<src_mac>[\w:]+)</mac>([\r\n]*)\s*</src>([\r\n]*)\s*<dst>([\r\n]*)\s*<ip>(?<dest_ip>[0-9.]+)</ip>([\r\n]*)\s*<mac>(?<dest_mac>[\w:]+)</mac>([\r\n]*)\s*<port>(?<dest_port>\d+)</port>([\r\n]*)\s*</dst>
0 Karma

TonyLeeVT
Builder

So others don't need to to make this change... We patched the Splunk for FireEye app to version 2.0.8 (making the change above) and pushed it to the Splunk apps site. I also created a howto for installation on Splunk v6 as a stopgap until the new app is completed. See the instructions below:

http://securitysynapse.com/2014/05/stopgap-splunk-for-fireeye-v2-app.html

Let us know if you have any issues in the mean time,
-Tony

0 Karma

TonyLeeVT
Builder

Try using the Universal fix before this one. I believe that should solve the problem no matter the version of FE OS. This will be left here just in case the other solution does not work for you.

Here is the fix for FireEye (pre-OS 7.1) appliances. This should work if your wMPS (NX) appliances are any version prior to 7.1. I will post a universal fix (that works for all appliances) if possible.

From the Splunk GUI:

App -> Manage Apps -> View Objects -> Search box: "EXTRACT-src-dst"

Click on the top hyperlink and replace the current contents with the following:

<src\svlan=\"\d+\"\><ip>(?<src_ip>[0-9.]+)</ip><host>(?<src_host>.*)</host><port>(?<src_port>\d+)</port><mac>(?<src_mac>[\w:]+)</mac></src><dst><ip>(?<dest_ip>[0-9.]+)</ip><mac>(?<dest_mac>[\w:]+)</mac><port>(?<dest_port>\d+)</port></dst>

Alternatively, you can also change this directly in the props.conf file:

Original:

EXTRACT-src-dst = \<src\svlan=\"\d+\"\>\<ip\>(?<src_ip>[0-9.]+)\</ip\>\<port\>(?<src_port>\d+)\</port\>\<mac\>(?<src_mac>[\w:]+)\</mac\>\</src\>\<dst\>\<ip\>(?<dest_ip>[0-9.]+)\</ip\>\<mac\>(?<dest_mac>[\w:]+)\</mac\>\<port\>(?<dest_port>\d+)\</port\>\</dst\>

Change to:

EXTRACT-src-dst = <src\svlan=\"\d+\"\><ip>(?<src_ip>[0-9.]+)</ip><host>(?<src_host>.*)</host><port>(?<src_port>\d+)</port><mac>(?<src_mac>[\w:]+)</mac></src><dst><ip>(?<dest_ip>[0-9.]+)</ip><mac>(?<dest_mac>[\w:]+)</mac><port>(?<dest_port>\d+)</port></dst>

Notice that the old regex did not take into account the tag which broke the parsing. Thanks.

Remember to restart splunkd after making this change: /opt/splunk/bin/splunk restart splunkd

TonyLeeVT
Builder

No problem Dave. Let me know if the Universal fix works for you. I tested it on FireEye OS 6.2 and 7.1 and it seems to be working.

0 Karma

ccsfdave
Builder

Thanks so much Tony, I am running 7.1.1 so if you get a chance for the universal fix, that would be great. In the mean time, I am going to see if the above works or can be manipulated to work for me.

0 Karma

ccsfdave
Builder

I am having the same issue with src_ip not being extracted. Did you ever figure this one out?

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

I'd check to make sure the fe_appliance, src_ip, malware_name and alert_id fields are being extracted. I'd do a search for index=fe sourcetype="fe_xml" and see if the field shows up on the left hand side.

Brian

ccsfdave
Builder

I am trying to extract the src_ip and malware_name fields and when I have Splunk write the regex in the "Interactive Field Extractor", it can't find the src_ip line probably because it is only pulling in the first 15 lines. Can I get this expanded? I could upload a screen shot if this is not clear...

0 Karma

TonyLeeVT
Builder

Keep in mind that there are different "names" within the events. For example:

One "name" contained within the tag:

Another "name" contained within the tag:

The root cause of the field extraction issue may be located in the props.conf or the transforms.conf file?

0 Karma

ccsfdave
Builder

Same with me, src_ip and malware_name (though name is) are not extracted. fe_appliance and alert_id are also fine

0 Karma

TonyLeeVT
Builder

Thanks Brian. It appears that src_ip and malware_name are not being extracted properly. fe_appliance and alert_id are fine.

0 Karma

wbkendall
Explorer

I couldn't post an event other than by editing the original post, but I now have one of the events in my post. I'm wondering if the format changed?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...