All Apps and Add-ons

Why is the indexing time lagging on email indexed by TA-Mailclient?

hcannon
Path Finder

We have had TA-Mailclient installed and working to index email via IMAP from a gmail account.

However we are noticing a lag between the time an email is received in the mailbox that is being monitored and the time the email is indexed, upwards of 40 minutes currently. What is odd is that the lag time has slowly increased over time - it was initially only a minute or two and has increased over the last few months to 30-40 minutes for each indexed email/event.

The app is installed on a search head that forwards to an indexer that has no known issues with indexing/parsing/etc, no other data sources are experiencing lag.

The input is scheduled to run every 100 seconds.
There are no errors in the logs for this app.
Splunk version is 7.0.1 and the app version is 1.3.0
Anyone else experiencing this issue or have any suggestions for further troubleshooting?

0 Karma
1 Solution

seunomosowon
Communicator

I have to try add some extra logic to continue reading using the message UID, instead of reading each message in the mailbox and comparing to the checkpoint. Might have time during the summer for that.

It’s currently reading all mails and skipping mails that have already been read, hence the increased lag time. It would be faster if it reads and deletes it.

View solution in original post

0 Karma

seunomosowon
Communicator

I have to try add some extra logic to continue reading using the message UID, instead of reading each message in the mailbox and comparing to the checkpoint. Might have time during the summer for that.

It’s currently reading all mails and skipping mails that have already been read, hence the increased lag time. It would be faster if it reads and deletes it.

0 Karma

Paul1896
Path Finder

One possibility to solve the problem of growing index time lag could be to use the KVstore function in future.

0 Karma

seunomosowon
Communicator

Yeah, I'm considering having a "retainFolder" option, and moving the read mails to that folder. I got your feature request for reading from other folders. I'll come back to you on that during the holidays 🙂

0 Karma

hcannon
Path Finder

Ah that makes sense and explains why the lag was slowly growing over time as this particular inbox grew in size. I updated the config to delete and this resolved the problem. thanks!

0 Karma

seunomosowon
Communicator

Are you having the app delete the emails as it is being read?

0 Karma

hcannon
Path Finder

Hi- nope right now we have the config set to read only.

0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...