How to troubleshoot why IMAP Mailbox is not indexing any email in our production Linux environment?

[sseifermann]

After the install and configuration of IMAP Mailbox app, it's not indexing any email. I was successful at getting this working in our lab environment, but not our production splunk environment.

Splunk is being run as root, so this shouldn't be a permission issue. I was able to run the debug command and it successfully returns mail in our production system.

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/IMAPmailbox/bin/get_imap_email.py --debug

What could I be missing? Does anyone know of any other troubleshooting steps to resolve this issue?

[hmozaffari]

I also experienced two challenges using IMAP App in corporate environments:

• Before installing and testing the App make sure mail server is supporting IMAP (ie Microsoft Exchange). You can test it with telnet: https://workaround.org/ispmail/lenny/test-fetching-with-imap-and-pop3
• You may have issues checking external mail boxes like Gmail when you are behind firewall.

[ragingwire · Accepted solution]

Yes it does. You will need to install the app on the search head. This will let you "see" the app in your splunk web. But you should not be running your input on this box at all. There is no index for it to save to. This could be your problem.

The README file has instructions for a distributed setup. There is a TA app loaded with this app. You will need to install that TA on the indexer. This is where your input will execute the get_imap_email.py script and thus be able to save it to your indexer.

I hope that helps.

[sseifermann]

ragingwire, that worked!

The TA add on needed to be installed and configured on the Indexer. See the release TA Release notes below.

Technology Add On (TA) for the IMAP Mailbox app

This TA is meant to be installed on Universal Forwarders, indexers, or other Forwarders if you are running a distributed Splunk design.

Install

1. Install this TA on all Forwarder(s) or Indexer(s) in the SPLUNK/etc/apps/ directory.
2. Install the IMAP Mailbox App on your Search head(s). Note: Do not configure the inputs.conf or imap.conf on the search head. In my case i configured these on the indexer.
   • Disable the input script.
   • Make sure that "disabled = true" for all of the inputs in the App under default/inputs.conf.
3. Enable inputs on ONE of your TAs.
   • Pick just one of the TA installs to be the collection point.
   • Copy defaults/imap.conf to local/imap.conf
   • Edit local/imap.conf with your correct server and user settings.
   • Copy defaults/inputs.conf to local/inputs.conf
   • Edit the inputs.conf file and enable the Unix or Windows script input.
   • Set "disabled=false" to the script input to enable.
   • Restart splunk. Note: this had to be done on the indexer and search head.

Note: Make sure that only ONE of the IMAP apps has the input script enabled. You will get email duplications if more than one is running.

[sseifermann]

I validated my settings and changed the following settings in the imap.conf
imapSearch = UNDELETED LARGER 1
deleteWhenDone = True
mimeTypes = text/richtext,text/plain,text/rtf

The emails are getting removed from the mailbox, but the imap mailbox app is still showing 0 indexed emails. Any other ideas?

[sseifermann]

Does it matter that our search head and indexer are on different servers?

[cpt12tech]

Try changing the search to return emails larger than 1 k. The following worked for me.

In the IMAP config file, I changed the search to:
imapSearch = UNDELETED LARGER 1

The theory is that email size is being combined with multiple emails. When there are a lot of emails, the size is larger than 300k (which is the default) and the emails are skipped and then deleted.

[cpt12tech]

Here are some of the changes I've done to get it to work:

1. double check the linux input is enabled in the inputs config file.
2. In the inputs config file, change the interval to a longer time than 5 min. I've set ours to 900 (15 min)
3. Turn caching off and have it delete mail after getting. You may not be able to do this, but it works almost all the time when I've told it to delete. I believe this is what has gotten it to work for me more than anything else.
4. If most of your emails are rich text, try adding text/rtf to the mime types.
5. change the rest timeout to 60 seconds

[ragingwire]

Make sure the script is enabled to run automatically

Look in local/imap.conf

[IMAP Configuration]
disabled = 0

Restart Splunk if you made any changes.
Or restart splunk just incase to see if that will trigger it to start.
You should see a log in index=_internal for the imap script if it is set to run.

[shivarpith]

Hi,

do we have to even make changes to get_imap_email.py file on the search head or comment out any specifics so that it connects to the addon ? i can see the script successfully connect to my mailbox but unale to link the forwarder to the search head.

[ragingwire]

no changes to the script needed. You should NOT run the script on your search head. It should be disabled there. You should run it from an indexer or forwarder if possible. There are instructions in the addon directory.