symantec Brightmail gateway- SBG field extraction

[thambisetty]

Hi,
I am getting the logs from SBG,but splunk couldnt able to index those logs. I need to index those logs. I did field extraction for first 3 fields are common in every event. The main problem is The next fields depend up on the 3rd field that is action. Now i want to extract those fields and i need to name it for search purpose. below are the example events.
1> jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

2> Jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e7-53b9ce61f951|IRCPTACTION|oceane-ias@orange.com|annotate

3>Jul 7 04:02:01 wipro-blr-out01 bmserver: 1404685921|cb5bdd57-f792c6d000001154-e7-53b9ce61f951|VERDICT|oceane-ias@orange.com|content_300|default|legal disclaimer

if u see above events fisrt 3 fields are common so i named it. Now i want name the rest of fields by using following search.

index=main sourcetype=ec_sbg_outbound action=accept----- now i will get all the events of action=accept it has only one field after action so i need to name for that field.
like that i have to do all the action types..
please help me
in advance thanks.....

[mshakeb]

Hi,

Try this add-on  for symantec Messaging gateway 

https://splunkbase.splunk.com/app/5215/

[Richfez]

Since you have the extraction for the first three fields, I suspect you can make the extraction for the remaining ones, you just need to know how to do that and make it work right.

In $splunkhome/etc/apps/myappname/local/transforms.conf (or wherever) you will need to create several REGEX statements. You have log lines like:

jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

So, use your REGEX you have for the first three fields, only don't extract the third field yet. Instead, include from the third field to the end of the line as something like "sbg_extra_info". BTW, I assume there's some "header" type of information that's not really "field1" - i.e. timestamp and so on. It doesn't matter for my explanation, I just mention it to prevent confusion below.

Some "pseudo regex" - meaning you may have to escape pipes, and honestly I just whipped it up so it's probably totally wrong, but it's close enough for our purposes:

[sbg-message-parse]
REGEX = ^(?<timestamp>[^ ]*\s+[^ ]*\s+[^ ]*)\s+(?<host>[^ ]*)\s+(?<some_other_field>[^ :]*)[:]\s+(?<field1>[^|]*)|(?<field2>[^ ]*)\s+(?<sbg_extra_info>.*)

But the important part was field1, field 2, then "everything else" as sbg_extra_info.

Now, also in that same transforms, create more stanzas, one for each of the type of service (ACCEPT, IRCPTACTION, VERDICT, etc...). Use "SOURCE_KEY = sbg_extra_info" to start by using that field for this extraction.

[sbg-extrainfo-accept-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>ACCEPT)|(?<accept_field1>[^|]*)(?<accept_field2>...

[sbg-extrainfo-ircptaction-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>IRCPTACTION)|(?<ircptaction_field1>[^|]*)(?<ircptaction_field2>...

Notice in each of those, I pull out the "service" (IRCPTACTION, ACCEPT... ) as well, then the rest of the REGEX just extracts whatever appropriate for the rest of the message. Add more fied extractions and stanzas as required.

Lastly, you have to call of these from props.conf. Order is important in that you have to pull out your sbg_extra_info FIRST. All the rest are on equal footing because there's no "nested" dependencies, just that one field needs to be created first. So, in $splunkhome/etc/apps/myappname/local/props.conf, call them all.

[mysourcetype]
REPORT-sbg_info = sbg-message-parse,sbg-extrainfo-accept-parse,sbg-extrainfo-ircptaction-parse,...

That should be it. I usually recommend getting the main sbg-message-parse right first, then proceeding with the rest. That way you can tweak each regex as a rex in a search directly and get it just right before committing it to your transforms.conf file.

[sorenmaigaard]

Hi

Did you ever find a way to parse these logs?

Best
Soren

[thambisetty]

Yes, I have created a TA which extracts only required event types.

[to4kawa]

https://community.splunk.com/t5/Splunk-Search/Splunk-TA-for-Symantec-Brightmail/td-p/477649

So you got a TA, right?

[alexling75]

Hi thambisetty,

Is it possible to share this TA ?

Thanks in advance.

Best regards,

Alex Ling

[thambisetty]

how can I share?

I will try to upload to Splunkbase.

[thambisetty · Accepted solution]

https://splunkbase.splunk.com/app/5181/

[thambisetty]

The Add-on is available now https://splunkbase.splunk.com/app/5181/

[mshakeb]

i cannot download the add-on

[alexling75]

Hi,

I am unable to see the page in splunkbase.

Can assist?

Thanks.

AL

[thambisetty]

@alexling75 

I have emailed you.

the one I uploaded to splunkbase is still pending for Splunkbase engineer review.

[alexling75]

Hi Balaji,

I am grateful for your kind sharing.

Thanks.

AL

[alexling75]

Hi thambisetty,

Can send to my email cwlingatyahoo.com?

Thanks.

Alex Ling

[alecdhuse]

I created this regex extraction, that extracts fields for the majority of Symantec Messaging Gateway's logs:

^<142>(?P<date>\w+\s+\d+)\s+(?P<time>[^ ]+)\s+(?P<server>\w+)\s+(?P<process_name>[a-z]+)\[(?P<process_number>\d+)[^ \n]* (?P<process_id>[^\|]+)\|(?P<message_id>[^\|]+)\|(?P<action>IRCPTACTION|VERDICT|UNTESTED|FIRED|SENDER|LOGICAL_IP|EHLO|MSG_SIZE|MSGID|SOURCE|SUBJECT|ORCPTS|TRACKERID|ATTACH|UNSCANNABLE|VIRUS|DELIVER|ACCEPT)(?:(?:(?<=ACCEPT|DELIVER|LOGICAL_IP)\|(?P<src>[^:\s]+)(?::(?P<port>[0-9]+))?(?:\|(?P<to>[^\s]+))?)|(?:(?<=FIRED|IRCPTACTION|ORCPTS|TRACKERID|UNTESTED|VERDICT)\|(?P<recipient>[^\s\|]+)(?:\|)?(?P<result>[a-z][^\|\s]+)?(?:\|(?P<result_2>[a-z][^\|]+))?(?:\|(?P<result_3>.+))?)|(?:(?<=SENDER)\|(?P<from>[^\s]+))|(?:(?<=MSG_SIZE)\|(?P<msg_size>\w+))|(?:(?<=SUBJECT)\|(?P<subject>.*))|(?:(?<=ATTACH)\|(?P<attachment>.+))|(?:(?<=UNSCANNABLE)\|(?P<reason>.+))|(?:(?<=VIRUS)\|(?P<virus_name>.+))|(?:(?<=EHLO)\|(?P<fqdn>.+)))?

I wrote a short blog post about it here: http://alec.dhuse.com/?p=217

[J9h0m0e8]

try, i used the logic:  

Error in 'SearchParser': Missing a search command before '^'. Error at position '151' of search query 'search source="tcp:3514" index="prueba" sourcetype...{snipped} {errorcontext = (?P<time>[^ ]+)\s+(?P}'.