Why is Microsoft 365 Defender Add-on for Splunk duplicating events?

[Azeemering]

Hello,

I have upgraded from the old defender app to the new Microsoft 365 Defender Add-on for Splunk.

I finally got it working after renewing secrets etc... but seems like there are a lot of duplicate events for each incident triggered. 
How can we get this too work properly? Can Splunk give proper support on this?  These small input apps are vital for a proper working of our SOC en Splunk ES environment. 

[thambisetty]

@VijaySrrie 

I haven't tested the one I have mentioned in my previous answer. 

I had followed the below document to integrate mcas logs to Splunk and its working fine. 
https://docs.microsoft.com/en-us/defender-cloud-apps/siem

Note:

Step 1: Set it up in the Defender for Cloud Apps portal
In the point 5. you need to enter your HF/indexer/syslog IP and port. 

if you run java jar on the host which you entered in step 1 and point 5 then you don't need to open any firewall connections, otherwise remember to open firewall connection from the host where you install java jar file and host you entered in step 1 and point 5. 

[VijaySrrie]

Hi @thambisetty 

I have opened the firewalls/proxy
In SIEM trace log file - no errors and logs related to connections are there
In MCAS portal it is showing as connected
In splunk, how should I check the logs?

Do we have any index or sourcetype?

[VijaySrrie]

 java jar files to be installed and run at the splunk end? or on the server end ? (which is gonna send logs to splunk)

[thambisetty]

you may try this https://splunkbase.splunk.com/app/5278/#/details

[VijaySrrie]

@thambisetty  how did you ingest MCAS logs into splunk?

The add-on which you have mentioned, is to provide value to the logs that are ingested via syslog. Is that correct?

You ingested logs using syslog collectors and then used above add-on to add values to the logs, is that correct?

[VijaySrrie]

Hi @jaxjohnny2000 @Azeemering @kristian_kolb @jconger @thambisetty 

Why we need this add-on? We have a requirement to ingest MCAS logs into splunk (salesforce logs flows into MCAS and those logs from MCAS to be ingested into splunk) Can I use the above add-on to achieve this?

Or should I use Syslog collectors to ingest MCAS logs into splunk?

[kristian_kolb · Accepted solution]

There is a slight bug(?) in the API calls where a comparison is made related to the checkpoint timestamp.

Specifically this is found on line 52 in

    TA-MS_Defender/bin/input_module_microsoft_defender_atp_alerts.py

where it says;

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+ge+%s" % (helper.get_arg("location"), query_date)

and I believe that the "ge" (>=) is the culprit here, and that its the reason for the last event from the previous run being indexed again. Changing it to "gt" (>) seemed to fix the problem. 

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+gt+%s" % (helper.get_arg("location"), query_date)

NB: The same thing is found in inputs_module_microsoft_365_defender_incidents.py in the same directory, but since I don't have that input, I haven't been able to verify that the behaviour is the same.

[jaxjohnny2000]

This appears to be corrected in version 1.3.0

[Azeemering]

Thank you...this works indeed for me too.  Very nice....well spotted!

I'm not exactly sure how to word it a 100% right...but I am disappointed that Splunk let's the community fix their released apps. 

[kristian_kolb]

Great to hear that it worked.

To be honest, though, the app is clearly labeled as "non-supported". It is not an official Splunk product.

I sent an email the other day to the person I believe to be the author to suggest this bug. No reply as yet, but I hope it will be fixed in the next release.

/Kristian

[jconger]

Can you share your inputs.conf?

Also, you can view the checkpoints by doing the following:

• Add the following lines to transforms.conf in the app:

[TA_Defender_checkpoint_lookup]
collection = TA_MS_Defender_checkpointer
external_type = kvstore
fields_list = _key, state

• Run the following search:

| inputlookup TA_Defender_checkpoint_lookup | eval key = _key

[Azeemering]

I also added a transforms.conf with your suggested settings and I can now see the checkpoint:

key state
atp_lastUpdateTime_MD_ATP_Acceptatie "2020-11-25T09:00:00Z"
atp_lastUpdateTime_MD_ATP_Productie "2020-11-30T09:37:04.08Z"

The funny thing is it seems to ingests the last event it finds again and again until there is a new alert. 

[Azeemering]

Hi, thanks for your quick reply!

[microsoft_defender_atp_alerts://MD_ATP_Acceptatie]
azure_app_account = ATP_Defender_Acceptatie
index = xxx_xxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
disabled = 0

[microsoft_defender_atp_alerts://MD_ATP_Productie]
azure_app_account = ATP_Defender_Productie
index = xxx_xxxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxx
disabled = 0

[Azeemering]

@jconger Anybody from Splunk who can help with this? The lack of support for app like this really is not a good development.

[Azeemering]

We are collecting the Alerts triggered in Windows Defender ATP from securitycenter.
This app was built by Splunk works, but not supported. But it should work right? Why release it otherwise?
What I have noticed during troubleshooting it indexes the same alerts every time the input runs on the schedule. So when a new event comes in and the input is running again the alert is indexed a second time. Seems like there is something wrong with checkpointing or something? 

[thambisetty]

@Azeemering 

The one you mentioned is neither supported by developer nor Splunk support.

what exactly are you looking for?

1. Are you looking for o365 defender TA  ?
2. do you want to  just collect Microsoft/windows defender agent logs installed on endpoints from cloud?

if you are looking for second  2  then you can try my TA link shared in my first response.

[Azeemering]

It's the https://splunkbase.splunk.com/app/4959/  Microsoft 365 Defender Add-on for Splunk

[thambisetty]

@Azeemering 

There are couple of TA around this subject. 

can you share TA URL?

I have also developed one