How can I tell whether TA-dmarc has pulled any content from an IMAP mailbox, and how can I search for it?

[ekzp]

Hi, installed TA-dmarc and it seems to be successfully accessing the mailbox and the Inbox/DMARC folder. It shows the number of messages found in the folder. But that's where it stops.

I am trying to determine whether it has actually pulled anything from the mailbox, and if so, how can I search for it.

Tried a number of searches. The best I could come up with is

index=* OR index=_* sourcetype=*dmarc*

Also tried

index=dmarc (sourcetype=dmarc OR sourcetype=dmarc:json)

and

index=* OR index=_* (sourcetype=dmarc OR sourcetype=dmarc:json)

inspired from https://github.com/aholzel/SA-dmarc as I intend to use SA-dmarc for dashboarding what TA-dmarc pulls down. None of these searches returned any results.

I even tried setting logging to debug, and I checked the content of %SPLUNKHOME%\log\splunk\ta_dmarc_splunc_imap.log - nothing useful.

So I have these two questions:
1. How can I tell whether TA-dmarc is actually pulling anything from the IMAP folder?
2. What search can I use that returns some results that I can plug into SA-dmarc for dashboarding? Effectively what I am asking is what search string needs to be entered as dmarc_log in the SA-dmark add-on's macros.conf file?

Thanks.

[richaatsnow]

@jorritf I am seeing all the messages (filter_seen_messages & processing of actual dmarc report mails) mentioned above by you. But don't see logs in a configured index.

How can I troubleshoot what's going on?

[jorritf]

Depending on the most recent DMARC report in the mailbox, you may need to adjust the search window to something like year-to-date.

[ekzp]

Update: seems to be working. The following query returned SPF and DKIM failures:

index=main (feedback{}.record.row.policy_evaluated.spf=fail OR feedback{}.record.row.policy_evaluated.dkim=fail)

Thanks for all the help,
Zoltan

[ekzp]

Hi Jorrit,

I can see everything up to "get_dmarc_message_bodies" including.

However I cannot see messages like "save_reports_from_message_bodies", "check_eligible_mimtype" and "write_part_to_file".

Does it mean that processing is incomplete?
Also, what search query can I use to pull some data?

Thanks.

[ekzp]

Thanks, I'll give it a try.

[jorritf]

Succesfull connections to my gmail dmarc mailbox look like this.
In DEBUG so sorry for the screenspam.

2019-02-17 19:35:05,971 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | Start processing imap server imap.gmail.com with use_ssl True
2019-02-17 19:35:06,060 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to imap.gmail.com
2019-02-17 19:35:08,047 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 343 messages in folder INBOX
2019-02-17 19:35:08,764 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 334 messages in folder INBOX match subject "Report domain:"

then look for filter_seen_messages:

19-02-17 19:35:09,403 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap   set([5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 1
7, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 
63, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 87, 88, 89, 90, 91, 92, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108
, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144
, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179
, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215
, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252
, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287
, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349])

and finally processing of actual dmarc report mails:

2019-02-17 19:35:09,403 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | Start processing 221 new messages of 334 on imap.gmail.com
2019-02-17 19:35:09,403 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 0 to 100
2019-02-17 19:35:22,082 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 100 to 200
2019-02-17 19:35:32,529 INFO pid=5027 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_message_bodies: getting messages 200 to 221
2019-02-17 19:35:38,666 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | save_reports_from_message_bodies: start non-multipart processing of msg uid  126
2019-02-17 19:35:38,666 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | check_eligible_mimtype: checking content-type application/zip of msg uid 126
2019-02-17 19:35:38,670 DEBUG pid=5027 tid=MainThread file=base_modinput.py:log_debug:286 | write_part_to_file: saved file /tmp/tmpwta3FW/google.com!nanowolk.nl!1526860800!1526947199.zip from uid 126

[richaatsnow]

@jorritf I am seeing all the messages mentioned here.

But dont see logs in the configured index

[jorritf]

Try to expand the timerange. The eventdate will be the date of the xml report. Depending on the report volume you may have to look back days, weeks or months.

[richaatsnow]

no luck with that

[ekzp]

The question still remains as aholzel's version errors out on installation.

How can I retrieve some reports with Jorrit Folmer's versio nof TA-dmarc?

Thanks.

[ekzp]

There seem to be two different add-ons by two developers, named identically: https://github.com/aholzel/TA-dmarc and https://github.com/jorritfolmer/TA-dmarc. It looks like I tried to combine SA-dmarc with the wrong TA-dmarc.