Why is the indexing time lagging on email indexed by TA-Mailclient?

[hcannon]

We have had TA-Mailclient installed and working to index email via IMAP from a gmail account.

However we are noticing a lag between the time an email is received in the mailbox that is being monitored and the time the email is indexed, upwards of 40 minutes currently. What is odd is that the lag time has slowly increased over time - it was initially only a minute or two and has increased over the last few months to 30-40 minutes for each indexed email/event.

The app is installed on a search head that forwards to an indexer that has no known issues with indexing/parsing/etc, no other data sources are experiencing lag.

The input is scheduled to run every 100 seconds.
There are no errors in the logs for this app.
Splunk version is 7.0.1 and the app version is 1.3.0
Anyone else experiencing this issue or have any suggestions for further troubleshooting?

[seunomosowon · Accepted solution]

I have to try add some extra logic to continue reading using the message UID, instead of reading each message in the mailbox and comparing to the checkpoint. Might have time during the summer for that.

It’s currently reading all mails and skipping mails that have already been read, hence the increased lag time. It would be faster if it reads and deletes it.

[Paul1896]

One possibility to solve the problem of growing index time lag could be to use the KVstore function in future.

[seunomosowon]

Yeah, I'm considering having a "retainFolder" option, and moving the read mails to that folder. I got your feature request for reading from other folders. I'll come back to you on that during the holidays 🙂

[hcannon]

Ah that makes sense and explains why the lag was slowly growing over time as this particular inbox grew in size. I updated the config to delete and this resolved the problem. thanks!

[seunomosowon]

Are you having the app delete the emails as it is being read?

[hcannon]

Hi- nope right now we have the config set to read only.