topic Re: Microsoft 365 Defender Add-on for Splunk duplicate events in All Apps and Add-ons

Why is Microsoft 365 Defender Add-on for Splunk duplicating events?

Azeemering — Wed, 23 Feb 2022 04:03:24 GMT

Hello,

I have upgraded from the old defender app to the new Microsoft 365 Defender Add-on for Splunk.

I finally got it working after renewing secrets etc... but seems like there are a lot of duplicate events for each incident triggered.
How can we get this too work properly? Can Splunk give proper support on this? These small input apps are vital for a proper working of our SOC en Splunk ES environment.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

thambisetty — Thu, 26 Nov 2020 10:56:54 GMT

@Azeemering

There are couple of TA around this subject.

can you share TA URL?

I have also developed one

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Thu, 26 Nov 2020 10:58:37 GMT

It's the https://splunkbase.splunk.com/app/4959/ Microsoft 365 Defender Add-on for Splunk

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

thambisetty — Thu, 26 Nov 2020 11:05:54 GMT

@Azeemering

The one you mentioned is neither supported by developer nor Splunk support.

what exactly are you looking for?

Are you looking for o365 defender TA ?
do you want to just collect Microsoft/windows defender agent logs installed on endpoints from cloud?

if you are looking for second 2 then you can try my TA link shared in my first response.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Thu, 26 Nov 2020 13:34:46 GMT

We are collecting the Alerts triggered in Windows Defender ATP from securitycenter.
This app was built by Splunk works, but not supported. But it should work right? Why release it otherwise?
What I have noticed during troubleshooting it indexes the same alerts every time the input runs on the schedule. So when a new event comes in and the input is running again the alert is indexed a second time. Seems like there is something wrong with checkpointing or something?

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Tue, 01 Dec 2020 13:37:53 GMT

@jconger Anybody from Splunk who can help with this? The lack of support for app like this really is not a good development.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

jconger — Tue, 01 Dec 2020 16:31:48 GMT

Can you share your inputs.conf?

Also, you can view the checkpoints by doing the following:

Add the following lines to transforms.conf in the app:

[TA_Defender_checkpoint_lookup] collection = TA_MS_Defender_checkpointer external_type = kvstore fields_list = _key, state

Run the following search:

| inputlookup TA_Defender_checkpoint_lookup | eval key = _key

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Tue, 01 Dec 2020 16:36:21 GMT

Hi, thanks for your quick reply!

[microsoft_defender_atp_alerts://MD_ATP_Acceptatie]
azure_app_account = ATP_Defender_Acceptatie
index = xxx_xxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
disabled = 0

[microsoft_defender_atp_alerts://MD_ATP_Productie]
azure_app_account = ATP_Defender_Productie
index = xxx_xxxxx
interval = 500
location = api-eu.securitycenter.microsoft.com
start_date = 2020-11-30T09:00:00Z
tenant_id = xxxxxxxxxxxxxxxxxxxxxx
disabled = 0

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Tue, 01 Dec 2020 16:45:49 GMT

I also added a transforms.conf with your suggested settings and I can now see the checkpoint:

key state
atp_lastUpdateTime_MD_ATP_Acceptatie "2020-11-25T09:00:00Z"
atp_lastUpdateTime_MD_ATP_Productie "2020-11-30T09:37:04.08Z"

The funny thing is it seems to ingests the last event it finds again and again until there is a new alert.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

kristian_kolb — Thu, 17 Dec 2020 08:42:43 GMT

There is a slight bug(?) in the API calls where a comparison is made related to the checkpoint timestamp.

Specifically this is found on line 52 in

TA-MS_Defender/bin/input_module_microsoft_defender_atp_alerts.py

where it says;

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+ge+%s" % (helper.get_arg("location"), query_date)

and I believe that the "ge" (>=) is the culprit here, and that its the reason for the last event from the previous run being indexed again. Changing it to "gt" (>) seemed to fix the problem.

atp_url = "https://%s/api/alerts?$expand=evidence&$filter=lastUpdateTime+gt+%s" % (helper.get_arg("location"), query_date)

NB: The same thing is found in inputs_module_microsoft_365_defender_incidents.py in the same directory, but since I don't have that input, I haven't been able to verify that the behaviour is the same.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Azeemering — Thu, 17 Dec 2020 13:12:35 GMT

Thank you...this works indeed for me too. Very nice....well spotted!

I'm not exactly sure how to word it a 100% right...but I am disappointed that Splunk let's the community fix their released apps.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

kristian_kolb — Thu, 17 Dec 2020 13:36:47 GMT

Great to hear that it worked.

To be honest, though, the app is clearly labeled as "non-supported". It is not an official Splunk product.

I sent an email the other day to the person I believe to be the author to suggest this bug. No reply as yet, but I hope it will be fixed in the next release.

/Kristian

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

jaxjohnny2000 — Tue, 05 Oct 2021 12:10:53 GMT

This appears to be corrected in version 1.3.0

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

VijaySrrie — Thu, 16 Dec 2021 03:27:32 GMT

Hi @jaxjohnny2000 @Azeemering @kristian_kolb @jconger @thambisetty

Why we need this add-on? We have a requirement to ingest MCAS logs into splunk (salesforce logs flows into MCAS and those logs from MCAS to be ingested into splunk) Can I use the above add-on to achieve this?

Or should I use Syslog collectors to ingest MCAS logs into splunk?

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

thambisetty — Sun, 06 Feb 2022 04:44:58 GMT

you may try this https://splunkbase.splunk.com/app/5278/#/details

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

VijaySrrie — Mon, 07 Feb 2022 03:55:56 GMT

@thambisetty how did you ingest MCAS logs into splunk?

The add-on which you have mentioned, is to provide value to the logs that are ingested via syslog. Is that correct?

You ingested logs using syslog collectors and then used above add-on to add values to the logs, is that correct?

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

thambisetty — Mon, 07 Feb 2022 07:41:18 GMT

@VijaySrrie

I haven't tested the one I have mentioned in my previous answer.

I had followed the below document to integrate mcas logs to Splunk and its working fine.
https://docs.microsoft.com/en-us/defender-cloud-apps/siem

Note:

Step 1: Set it up in the Defender for Cloud Apps portal
In the point 5. you need to enter your HF/indexer/syslog IP and port.

if you run java jar on the host which you entered in step 1 and point 5 then you don't need to open any firewall connections, otherwise remember to open firewall connection from the host where you install java jar file and host you entered in step 1 and point 5.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

VijaySrrie — Mon, 07 Feb 2022 07:52:49 GMT

java jar files to be installed and run at the splunk end? or on the server end ? (which is gonna send logs to splunk)

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

VijaySrrie — Wed, 23 Feb 2022 00:30:39 GMT

Hi @thambisetty

I have opened the firewalls/proxy
In SIEM trace log file - no errors and logs related to connections are there
In MCAS portal it is showing as connected
In splunk, how should I check the logs?

Do we have any index or sourcetype?

topic Re: Microsoft 365 Defender Add-on for Splunk duplicate events in All Apps and Add-ons

Why is Microsoft 365 Defender Add-on for Splunk duplicating events?

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Note:

Step 1: Set it up in the Defender for Cloud Apps portalIn the point 5. you need to enter your HF/indexer/syslog IP and port.

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Re: Microsoft 365 Defender Add-on for Splunk duplicate events

Step 1: Set it up in the Defender for Cloud Apps portal
In the point 5. you need to enter your HF/indexer/syslog IP and port.