All Apps and Add-ons

Why is Microsoft 365 Defender Add-on for Splunk duplicating events?

Azeemering
Builder

Hello,

I have upgraded from the old defender app to the new Microsoft 365 Defender Add-on for Splunk.

I finally got it working after renewing secrets etc... but seems like there are a lot of duplicate events for each incident triggered. 
How can we get this too work properly? Can Splunk give proper support on this?  These small input apps are vital for a proper working of our SOC en Splunk ES environment. 

Labels (4)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

There is a slight bug(?) in the API calls where a comparison is made related to the checkpoint timestamp.

Specifically this is found on line 52 in

    TA-MS_Defender/bin/input_module_microsoft_defender_atp_alerts.py

where it says;

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+ge+%s" % (helper.get_arg("location"), query_date)

and I believe that the "ge" (>=) is the culprit here, and that its the reason for the last event from the previous run being indexed again. Changing it to "gt" (>) seemed to fix the problem. 

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+gt+%s" % (helper.get_arg("location"), query_date)


 

NB: The same thing is found in inputs_module_microsoft_365_defender_incidents.py in the same directory, but since I don't have that input, I haven't been able to verify that the behaviour is the same.

 

View solution in original post

thambisetty
SplunkTrust
SplunkTrust

@VijaySrrie 

I haven't tested the one I have mentioned in my previous answer. 


I had followed the below document to integrate mcas logs to Splunk and its working fine. 
https://docs.microsoft.com/en-us/defender-cloud-apps/siem

Note:

Step 1: Set it up in the Defender for Cloud Apps portal
In the point 5. you need to enter your HF/indexer/syslog IP and port. 

if you run java jar on the host which you entered in step 1 and point 5 then you don't need to open any firewall connections, otherwise remember to open firewall connection from the host where you install java jar file and host you entered in step 1 and point 5. 

 

————————————
If this helps, give a like below.

VijaySrrie
Builder

Hi @thambisetty 

I have opened the firewalls/proxy
In SIEM trace log file - no errors and logs related to connections are there
In MCAS portal it is showing as connected
In splunk, how should I check the logs?

Do we have any index or sourcetype?

Tags (1)
0 Karma

VijaySrrie
Builder

 java jar files to be installed and run at the splunk end? or on the server end ? (which is gonna send logs to splunk)

0 Karma

thambisetty
SplunkTrust
SplunkTrust

you may try this https://splunkbase.splunk.com/app/5278/#/details

————————————
If this helps, give a like below.
0 Karma

VijaySrrie
Builder

@thambisetty  how did you ingest MCAS logs into splunk?

The add-on which you have mentioned, is to provide value to the logs that are ingested via syslog. Is that correct?

You ingested logs using syslog collectors and then used above add-on to add values to the logs, is that correct?

0 Karma

VijaySrrie
Builder

Hi @jaxjohnny2000 @Azeemering @kristian_kolb @jconger @thambisetty 

Why we need this add-on? We have a requirement to ingest MCAS logs into splunk (salesforce logs flows into MCAS and those logs from MCAS to be ingested into splunk) Can I use the above add-on to achieve this?

Or should I use Syslog collectors to ingest MCAS logs into splunk?

0 Karma

kristian_kolb
Ultra Champion

There is a slight bug(?) in the API calls where a comparison is made related to the checkpoint timestamp.

Specifically this is found on line 52 in

    TA-MS_Defender/bin/input_module_microsoft_defender_atp_alerts.py

where it says;

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+ge+%s" % (helper.get_arg("location"), query_date)

and I believe that the "ge" (>=) is the culprit here, and that its the reason for the last event from the previous run being indexed again. Changing it to "gt" (>) seemed to fix the problem. 

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+gt+%s" % (helper.get_arg("location"), query_date)


 

NB: The same thing is found in inputs_module_microsoft_365_defender_incidents.py in the same directory, but since I don't have that input, I haven't been able to verify that the behaviour is the same.

 

jaxjohnny2000
Builder

This appears to be corrected in version 1.3.0

0 Karma

Azeemering
Builder

Thank you...this works indeed for me too.  Very nice....well spotted!

I'm not exactly sure how to word it a 100% right...but I am disappointed that Splunk let's the community fix their released apps. 

0 Karma

kristian_kolb
Ultra Champion

Great to hear that it worked.

To be honest, though, the app is clearly labeled as "non-supported". It is not an official Splunk product.

I sent an email the other day to the person I believe to be the author to suggest this bug. No reply as yet, but I hope it will be fixed in the next release.

/Kristian

0 Karma

jconger
Splunk Employee
Splunk Employee

Can you share your inputs.conf?

Also, you can view the checkpoints by doing the following:

  • Add the following lines to transforms.conf in the app:
[TA_Defender_checkpoint_lookup]
collection = TA_MS_Defender_checkpointer
external_type = kvstore
fields_list = _key, state

 

  • Run the following search:
| inputlookup TA_Defender_checkpoint_lookup | eval key = _key

 

0 Karma

Azeemering
Builder

I also added a transforms.conf with your suggested settings and I can now see the checkpoint:

key state
atp_lastUpdateTime_MD_ATP_Acceptatie "2020-11-25T09:00:00Z"
atp_lastUpdateTime_MD_ATP_Productie "2020-11-30T09:37:04.08Z"

The funny thing is it seems to ingests the last event it finds again and again until there is a new alert. 

 

0 Karma

Azeemering
Builder

Hi, thanks for your quick reply!

[microsoft_defender_atp_alerts://MD_ATP_Acceptatie]
azure_app_account = ATP_Defender_Acceptatie
index = xxx_xxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
disabled = 0

[microsoft_defender_atp_alerts://MD_ATP_Productie]
azure_app_account = ATP_Defender_Productie
index = xxx_xxxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxx
disabled = 0

0 Karma

Azeemering
Builder

@jconger Anybody from Splunk who can help with this? The lack of support for app like this really is not a good development.

0 Karma

Azeemering
Builder

We are collecting the Alerts triggered in Windows Defender ATP from securitycenter.
This app was built by Splunk works, but not supported. But it should work right? Why release it otherwise?
What I have noticed during troubleshooting it indexes the same alerts every time the input runs on the schedule. So when a new event comes in and the input is running again the alert is indexed a second time. Seems like there is something wrong with checkpointing or something? 

0 Karma

thambisetty
SplunkTrust
SplunkTrust

@Azeemering 

The one you mentioned is neither supported by developer nor Splunk support.

what exactly are you looking for?

  1. Are you looking for o365 defender TA  ?
  2. do you want to  just collect Microsoft/windows defender agent logs installed on endpoints from cloud?

if you are looking for second  2  then you can try my TA link shared in my first response.

————————————
If this helps, give a like below.
0 Karma

Azeemering
Builder

It's the https://splunkbase.splunk.com/app/4959/  Microsoft 365 Defender Add-on for Splunk

0 Karma

thambisetty
SplunkTrust
SplunkTrust

@Azeemering 

There are couple of TA around this subject. 

can you share TA URL?

I have also developed one

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...