topic Re: Index emails to splunk in Getting Data In

Index emails to splunk

neeravmathur — Thu, 14 Oct 2021 09:09:29 GMT

Hi Guys,

We have a requirement where we need to index emails to be ingested into splunk. I know a couple of apps are out there but I could not get them working...also not sure how to setup/request a mail account for splunk specifically for this purpose like what all settings should be applied etc.

I am a novice as far as mail settings are concerned, so can someone take some time and help me out here and be as detailed as possible...We are using Splunk 8.0.0

Thanks,

Neerav

Re: Index emails to splunk

PickleRick — Thu, 14 Oct 2021 09:19:27 GMT

Firstly, you should ask yourself what do you mean by "index emails". You want to have raw email headers in your splunk indexes? Email bodies? Some form of a header "extract"?

Secondly - where do you have or can have the emails that you want to perform indexing on? Do you have some account in Exchange that you want to pull the emails from? Or do you want to do indexing on all emails coming into your postfix/exim/whatever server? Do you want to process emails for a particular user or emails for all users?

Answers to those two questions should point you towards a reasonable solution.

Re: Index emails to splunk

neeravmathur — Thu, 14 Oct 2021 10:26:42 GMT

@PickleRick Thanks for your prompt response....

We have Oracle OEM sending out error messages via emails to some users and we need a splunk specific account which can be added on this mail. So we would need to ingest actual mail body as they contain info about Oracle alerts.

These users have domain/NT account email IDs (O365).

Need help regarding what app to use compatible with splunk 8.0.0 and some details of how to set it up.

xxxxxxxxxxxxxxxxxxxSAMPLE MAILxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

From: 13 C Oracle EM Notifications
Sent: Tuesday, September 28, 2021 10:22 AM
To: X.Y@abc.com
Subject: EM Event: Critical:bwprod - SQL running For Long TIme: UserInfo = Inst: 3\,SID:2338 \, OSUSER:xxxxxx-yyyy \, machine:US*******\, sql_id:2ptaaaaaaaaaaaaar Time(in mins) = 76

Host=us**********
Target type=Cluster Database
Target name=bwprod2
Categories=Performance
Message=SQL running For Long TIme: UserInfo = Inst: 3\,SID:2338 \, OSUSER:xxxxxx-yyyy \, machine:US*******\, sql_id:2ptaaaaaaaaaaaaar Time(in mins) = 76
Severity=Critical
Event reported time=Sep 28, 2021 10:22:13 AM EDT
Operating System=Linux
Platform=x86_64
Associated Incident Id=3777777
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=ME$Long_running_queries:Elapsed_Time_mins
Metric Group=ME$Long_running_queries
Metric=Elapsed_Time_mins
Metric value=76
Key Value=Inst: 3\,SID:2338 \, OSUSER:xxxxxx-yyyy \, machine:US*******\, sql_id:2ptaaaaaaaaaaaaar Time(in mins) = 76
Key Column 1=Userinfo
Rule Name=xxxxxxxxxxxxxxx
Rule Owner=xyxyxyxyx
Update Details:
SQL running For Long TIme: UserInfo = Inst: 3\,SID:2338 \, OSUSER:xxxxxx-yyyy \, machine:US*******\, sql_id:2ptaaaaaaaaaaaaar Time(in mins) = 76
Incident created by rule (Name = Incident management rule set for all targets, Create incident for critical metric alerts [System generated rule]).

Thanks,

Neerav

Re: Index emails to splunk

PickleRick — Thu, 14 Oct 2021 12:34:50 GMT

OK. I can tell you what I would do.

Firstly, you have to make sure that your emails are delivered to a specific account. Whether you can do it on the sender's side by defining additional recipient or you have to add additional rules in your mail server - that's outside of the scope of this forum and it's up to your admins 🙂

The rest depends heavily on your email infrastructure. I don't think there's any ready-made app for pulling emails from any pop3 or imap service so it all have to be written from scratch.

There are generally two approaches you can take:

deliver emails for this account to a specific machine on which you'll run a script be means of procmail or similar software, which will extract the body from the mail message, possibly filter it and convert a little and finally either write to a file from which it would be picked up by UF or send it to a splunk input (possibly HEC).
have a script run on schedule (cron on linux machine, task scheduler on windows) that will connect to your email account by means of POP3, IMAP, MAPI or any other mechanism that you use in your company, retrieve new mails, transform them, filter and write to a file or send to an input.

Unfortunately, since it's a very uncommon mode of providing the events for splunk, I'm afraid you'll have to write everything from scratch.

Are you sure there's no other way of delivering those events to splunk? Some log files? Syslog?

Re: Index emails to splunk

neeravmathur — Mon, 01 Nov 2021 14:11:38 GMT

Hi @PickleRick ,

After a long discussion we have decided to use an IMAP based shared mail box where splunk user will have access and use the app-TA-mailclient to ingest the mails. Will let you know about the progress.

Thanks