topic Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it? in All Apps and Add-ons

How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Tue, 29 Sep 2020 22:55:41 GMT

Hi, installed TA-dmarc and it seems to be successfully accessing the mailbox and the Inbox/DMARC folder. It shows the number of messages found in the folder. But that's where it stops.

I am trying to determine whether it has actually pulled anything from the mailbox, and if so, how can I search for it.

Tried a number of searches. The best I could come up with is

index=* OR index=_* sourcetype=*dmarc*

Also tried

index=dmarc (sourcetype=dmarc OR sourcetype=dmarc:json)

and

index=* OR index=_* (sourcetype=dmarc OR sourcetype=dmarc:json)

inspired from https://github.com/aholzel/SA-dmarc as I intend to use SA-dmarc for dashboarding what TA-dmarc pulls down. None of these searches returned any results.

I even tried setting logging to debug, and I checked the content of %SPLUNKHOME%\log\splunk\ta_dmarc_splunc_imap.log - nothing useful.

So I have these two questions:
1. How can I tell whether TA-dmarc is actually pulling anything from the IMAP folder?
2. What search can I use that returns some results that I can plug into SA-dmarc for dashboarding? Effectively what I am asking is what search string needs to be entered as dmarc_log in the SA-dmark add-on's macros.conf file?

Thanks.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Fri, 25 Jan 2019 07:24:03 GMT

There seem to be two different add-ons by two developers, named identically: https://github.com/aholzel/TA-dmarc and https://github.com/jorritfolmer/TA-dmarc. It looks like I tried to combine SA-dmarc with the wrong TA-dmarc.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Tue, 29 Jan 2019 05:36:09 GMT

The question still remains as aholzel's version errors out on installation.

How can I retrieve some reports with Jorrit Folmer's versio nof TA-dmarc?

Thanks.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

jorritf — Tue, 29 Sep 2020 23:17:40 GMT

Succesfull connections to my gmail dmarc mailbox look like this.
In DEBUG so sorry for the screenspam.

2019-02-17 19:35:05,971 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | Start processing imap server imap.gmail.com with use_ssl True
2019-02-17 19:35:06,060 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to imap.gmail.com
2019-02-17 19:35:08,047 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 343 messages in folder INBOX
2019-02-17 19:35:08,764 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 334 messages in folder INBOX match subject "Report domain:"

then look for filter_seen_messages:

19-02-17 19:35:09,403 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap   set([5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 1
7, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 
63, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 87, 88, 89, 90, 91, 92, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108
, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144
, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179
, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215
, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252
, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287
, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349])

and finally processing of actual dmarc report mails:

2019-02-17 19:35:09,403 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | Start processing 221 new messages of 334 on imap.gmail.com
2019-02-17 19:35:09,403 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 0 to 100
2019-02-17 19:35:22,082 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 100 to 200
2019-02-17 19:35:32,529 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 200 to 221
2019-02-17 19:35:38,666 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | save_reports_from_message_bodies: start non-multipart processing of msg uid  126
2019-02-17 19:35:38,666 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | check_eligible_mimtype: checking content-type application/zip of msg uid 126
2019-02-17 19:35:38,670 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | write_part_to_file: saved file /tmp/tmpwta3FW/google.com!nanowolk.nl!1526860800!1526947199.zip from uid 126

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Mon, 25 Feb 2019 06:54:26 GMT

Thanks, I'll give it a try.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Tue, 29 Sep 2020 23:29:05 GMT

Hi Jorrit,

I can see everything up to "get_dmarc_message_bodies" including.

However I cannot see messages like "save_reports_from_message_bodies", "check_eligible_mimtype" and "write_part_to_file".

Does it mean that processing is incomplete?
Also, what search query can I use to pull some data?

Thanks.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

ekzp — Tue, 29 Sep 2020 23:29:07 GMT

Update: seems to be working. The following query returned SPF and DKIM failures:

index=main (feedback{}.record.row.policy_evaluated.spf=fail OR feedback{}.record.row.policy_evaluated.dkim=fail)

Thanks for all the help,
Zoltan

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

richaatsnow — Wed, 30 Sep 2020 05:33:47 GMT

@jorritf I am seeing all the messages (filter_seen_messages & processing of actual dmarc report mails) mentioned above by you. But don't see logs in a configured index.

How can I troubleshoot what's going on?

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

richaatsnow — Tue, 26 May 2020 18:23:25 GMT

@jorritf I am seeing all the messages mentioned here.

But dont see logs in the configured index

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

jorritf — Tue, 26 May 2020 18:39:06 GMT

Try to expand the timerange. The eventdate will be the date of the xml report. Depending on the report volume you may have to look back days, weeks or months.

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

richaatsnow — Tue, 26 May 2020 18:46:18 GMT

no luck with that

Re: How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

jorritf — Thu, 01 Oct 2020 07:25:47 GMT

Depending on the most recent DMARC report in the mailbox, you may need to adjust the search window to something like year-to-date.