Search Proofpoint Logs

[RB5]

Hi, I was hoping to get help for a search. I haven't had much time to spend on it so I apoligize for not trying harder 1st.

I've started out with below, but both searches only return 2 results, even though there are over 1K log entries of the same format.

index=xyz | transaction startswith="cmd=connect" endswith="cmd=disconnect" 

OR:

index=xyz * | transaction s,m maxspan=301s startswith="mod=session cmd=connect" endswith="mod=session cmd=disconnect"

I want to pull items such as value=abc@xyz.com where have cmd=env_rcpt, value=uvh@gmail.com where have cmd=env_from and pull 'subject' and various scores,
like '3' from: suspectscore=3

The log entries are of the format below.

Thanks

[2011-10-23 16:05:59.502387 +0000] rprt s=10kch03n9t mod=session cmd=connect ip=209.85.210.182  perlwait=0.085
[2011-10-23 16:06:26.251606 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_from value=uvh@gmail.com qid=p9NG5xMt010615  ip=209.85.210.182
[2011-10-23 16:06:26.405437 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_rcpt r=1 value=abc@xyz.com verified= routes=
[2011-10-23 16:06:26.875486 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=spam cmd=run score=0 spamscore=0 ipscore=0 suspectscore=3 phishscore=0 bulkscore=0 adultscore=0 duration=0.091
[2011-10-23 16:06:26.879828 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=msg module=spf rule=pass action=continue attachments=0 rcpts=1 subject="Hi" spamscore=0
[2011-10-23 16:06:56.927722 +0000] rprt s=10kch03n9t mod=session cmd=disconnect module= rule= action= helo=mail-iy0-f182.google.com msgs=3 rcpts=3 routes= duration=1.119 elapsed=57.43

[2011-10-23 17:05:59.502387 +0000] rprt s=10kch03n9t mod=session cmd=connect ip=209.85.210.182  perlwait=0.085
[2011-10-23 17:06:26.251606 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_from value=xyz@hotmail.com qid=p9NG5xMt010615  ip=209.85.210.182
[2011-10-23 17:06:26.405437 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_rcpt r=1 value=123@xyz.com verified= routes=
[2011-10-23 17:06:26.875486 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=spam cmd=run score=0 spamscore=0 ipscore=0 suspectscore=3 phishscore=0 bulkscore=0 adultscore=0 duration=0.091
[2011-10-23 17:06:26.879828 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=msg module=spf rule=pass action=continue attachments=0 rcpts=1 subject="Hi" spamscore=0
[2011-10-23 17:06:56.927722 +0000] rprt s=10kch03n9t mod=session cmd=disconnect module= rule= action= helo=mail-iy0-f182.google.com msgs=3 rcpts=3 routes= duration=1.119 elapsed=57.43

[2011-10-23 18:05:59.502387 +0000] rprt s=10kch03n9t mod=session cmd=connect ip=209.85.210.182  perlwait=0.085
[2011-10-23 18:06:26.251606 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_from value=123@gmail.com qid=p9NG5xMt010615  ip=209.85.210.182
[2011-10-23 18:06:26.405437 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=env_rcpt r=1 value=xxtt@xyz.com verified= routes=
[2011-10-23 18:06:26.875486 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=spam cmd=run score=0 spamscore=0 ipscore=0 suspectscore=3 phishscore=0 bulkscore=0 adultscore=0 duration=0.091
[2011-10-23 18:06:26.879828 +0000] rprt s=10kch03n9t m=3 x=p9NG5xMt010615 mod=mail cmd=msg module=spf rule=pass action=continue attachments=0 rcpts=1 subject="Hi" spamscore=0
[2011-10-23 18:06:56.927722 +0000] rprt s=10kch03n9t mod=session cmd=disconnect module= rule= action= helo=mail-iy0-f182.google.com msgs=3 rcpts=3 routes= duration=1.119 elapsed=57.43

[eckolp2003]

Proofpoint now has a beta app that will allow you report on and visualze your Proofpoint Protection Server and TAP data! Check out the new app here:

https://splunkbase.splunk.com/app/3727/#/details

Be sure to follow the instructions listed in the details to get all the needed TA's etc that the app needs to work correctly.

There are pre-built dashboards to aid in searching for message events.

[RB5]

Thanks Kristian, I'll look into your recommendations. To answer your questions:

yes, it is the same set of events, all emails, s=ssssss is a session identifier, m=mm is a particulater email/message. There can be (and are) multiple emails associated with a session. There are a lot more log entries per each email, but I condensed it to the fields I want to pull.
Thanks again.

[kristian_kolb]

You are aware of what the transaction command is supposed to be doing, right? It will bundle all events matching the conditions into transactions. The result will be a new multi-line event for each transaction.

In your first query, you'd probably get the wrong transaction-events, as the only condition is connect/disconnect - regardless of any other data in the events. It may be ok for something very sequential in nature, where sessions do not intermingle. But for some kind of webserver log, it would be meaningless.

So, what are the events you have listed? It looks like it the same set of events repeated 3 times.

Is s some kind of session identifier? (in respect to the sessions of the spamfilter(?))
What is m? Why is that important?
It seems like x is present on all lines that you're interested in.

There is a bit of a problem when the logging application write logs in the style of key=<key> value=<value>, instead of <key>=<value>, since the field value will be ambiguous. For this exercise, we'll use rex to do the extraction as part of the search.

index=xyz 
| rex "env_from\s+value=(?<sender>\S+)" 
| rex "env_rcpt\s+r=\d+value=(?<receiver>\S+)"
| transaction s maxspan=5m

This should build the transaction that you're looking for. However, there might be a simpler way by using stats. stats is cheaper than transaction in terms of execution costs. Modify the query below to fit your own needs.

index=xyz 
| rex "env_from\s+value=(?<sender>\S+)" 
| rex "env_rcpt\s+r=\d+value=(?<receiver>\S+)"
| stats list(sender) as Sender values(receiver) as Receivers first(subject) as Subject max(suspectscore) as SuspectScore by s
| where SuspectScore>2

Hope this helps,

Kristian