topic Re: symantec Brightmail gateway- SBG field extraction in Splunk Search

symantec Brightmail gateway- SBG field extraction

thambisetty — Mon, 28 Sep 2020 17:00:12 GMT

Hi,
I am getting the logs from SBG,but splunk couldnt able to index those logs. I need to index those logs. I did field extraction for first 3 fields are common in every event. The main problem is The next fields depend up on the 3rd field that is action. Now i want to extract those fields and i need to name it for search purpose. below are the example events.
1> jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

2> Jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e7-53b9ce61f951|IRCPTACTION|oceane-ias@orange.com|annotate

if u see above events fisrt 3 fields are common so i named it. Now i want name the rest of fields by using following search.

index=main sourcetype=ec_sbg_outbound action=accept----- now i will get all the events of action=accept it has only one field after action so i need to name for that field.
like that i have to do all the action types..
please help me
in advance thanks.....

Re: symantec Brightmail gateway- SBG field extraction

sorenmaigaard — Fri, 02 Jan 2015 23:13:10 GMT

Did you ever find a way to parse these logs?

Best
Soren

Re: symantec Brightmail gateway- SBG field extraction

Richfez — Tue, 29 Sep 2020 07:38:00 GMT

Since you have the extraction for the first three fields, I suspect you can make the extraction for the remaining ones, you just need to know how to do that and make it work right.

In $splunkhome/etc/apps/myappname/local/transforms.conf (or wherever) you will need to create several REGEX statements. You have log lines like:

jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

So, use your REGEX you have for the first three fields, only don't extract the third field yet. Instead, include from the third field to the end of the line as something like "sbg_extra_info". BTW, I assume there's some "header" type of information that's not really "field1" - i.e. timestamp and so on. It doesn't matter for my explanation, I just mention it to prevent confusion below.

Some "pseudo regex" - meaning you may have to escape pipes, and honestly I just whipped it up so it's probably totally wrong, but it's close enough for our purposes:

[sbg-message-parse]
REGEX = ^(?<timestamp>[^ ]*\s+[^ ]*\s+[^ ]*)\s+(?<host>[^ ]*)\s+(?<some_other_field>[^ :]*)[:]\s+(?<field1>[^|]*)|(?<field2>[^ ]*)\s+(?<sbg_extra_info>.*)

But the important part was field1, field 2, then "everything else" as sbg_extra_info.

Now, also in that same transforms, create more stanzas, one for each of the type of service (ACCEPT, IRCPTACTION, VERDICT, etc...). Use "SOURCE_KEY = sbg_extra_info" to start by using that field for this extraction.

[sbg-extrainfo-accept-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>ACCEPT)|(?<accept_field1>[^|]*)(?<accept_field2>...

[sbg-extrainfo-ircptaction-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>IRCPTACTION)|(?<ircptaction_field1>[^|]*)(?<ircptaction_field2>...

Notice in each of those, I pull out the "service" (IRCPTACTION, ACCEPT... ) as well, then the rest of the REGEX just extracts whatever appropriate for the rest of the message. Add more fied extractions and stanzas as required.

Lastly, you have to call of these from props.conf. Order is important in that you have to pull out your sbg_extra_info FIRST. All the rest are on equal footing because there's no "nested" dependencies, just that one field needs to be created first. So, in $splunkhome/etc/apps/myappname/local/props.conf, call them all.

[mysourcetype]
REPORT-sbg_info = sbg-message-parse,sbg-extrainfo-accept-parse,sbg-extrainfo-ircptaction-parse,...

That should be it. I usually recommend getting the main sbg-message-parse right first, then proceeding with the rest. That way you can tweak each regex as a rex in a search directly and get it just right before committing it to your transforms.conf file.

Re: symantec Brightmail gateway- SBG field extraction

alecdhuse — Thu, 22 Sep 2016 15:01:38 GMT

I created this regex extraction, that extracts fields for the majority of Symantec Messaging Gateway's logs:

^<142>(?P<date>\w+\s+\d+)\s+(?P<time>[^ ]+)\s+(?P<server>\w+)\s+(?P<process_name>[a-z]+)\[(?P<process_number>\d+)[^ \n]* (?P<process_id>[^\|]+)\|(?P<message_id>[^\|]+)\|(?P<action>IRCPTACTION|VERDICT|UNTESTED|FIRED|SENDER|LOGICAL_IP|EHLO|MSG_SIZE|MSGID|SOURCE|SUBJECT|ORCPTS|TRACKERID|ATTACH|UNSCANNABLE|VIRUS|DELIVER|ACCEPT)(?:(?:(?<=ACCEPT|DELIVER|LOGICAL_IP)\|(?P<src>[^:\s]+)(?::(?P<port>[0-9]+))?(?:\|(?P<to>[^\s]+))?)|(?:(?<=FIRED|IRCPTACTION|ORCPTS|TRACKERID|UNTESTED|VERDICT)\|(?P<recipient>[^\s\|]+)(?:\|)?(?P<result>[a-z][^\|\s]+)?(?:\|(?P<result_2>[a-z][^\|]+))?(?:\|(?P<result_3>.+))?)|(?:(?<=SENDER)\|(?P<from>[^\s]+))|(?:(?<=MSG_SIZE)\|(?P<msg_size>\w+))|(?:(?<=SUBJECT)\|(?P<subject>.*))|(?:(?<=ATTACH)\|(?P<attachment>.+))|(?:(?<=UNSCANNABLE)\|(?P<reason>.+))|(?:(?<=VIRUS)\|(?P<virus_name>.+))|(?:(?<=EHLO)\|(?P<fqdn>.+)))?

I wrote a short blog post about it here: http://alec.dhuse.com/?p=217

Re: symantec Brightmail gateway- SBG field extraction

thambisetty — Sun, 26 Jul 2020 05:49:23 GMT

Yes, I have created a TA which extracts only required event types.

Re: symantec Brightmail gateway- SBG field extraction

alexling75 — Wed, 12 Aug 2020 02:54:38 GMT

Hi thambisetty,

Is it possible to share this TA ?

Thanks in advance.

Best regards,

Alex Ling

Re: symantec Brightmail gateway- SBG field extraction

thambisetty — Wed, 12 Aug 2020 05:40:12 GMT

how can I share?

I will try to upload to Splunkbase.

Re: symantec Brightmail gateway- SBG field extraction

alexling75 — Wed, 12 Aug 2020 05:43:39 GMT

Hi thambisetty,

Can send to my email cwlingatyahoo.com?

Thanks.

Alex Ling

Re: symantec Brightmail gateway- SBG field extraction

thambisetty — Wed, 12 Aug 2020 07:12:01 GMT

https://splunkbase.splunk.com/app/5181/

Re: symantec Brightmail gateway- SBG field extraction

to4kawa — Wed, 12 Aug 2020 07:25:20 GMT

https://community.splunk.com/t5/Splunk-Search/Splunk-TA-for-Symantec-Brightmail/td-p/477649

So you got a TA, right?

Re: symantec Brightmail gateway- SBG field extraction

alexling75 — Wed, 12 Aug 2020 07:28:43 GMT

Hi,

I am unable to see the page in splunkbase.

Can assist?

Thanks.

Re: symantec Brightmail gateway- SBG field extraction

thambisetty — Wed, 12 Aug 2020 08:21:06 GMT

@alexling75

I have emailed you.

the one I uploaded to splunkbase is still pending for Splunkbase engineer review.

Re: symantec Brightmail gateway- SBG field extraction

alexling75 — Wed, 12 Aug 2020 08:27:40 GMT

Hi Balaji,

I am grateful for your kind sharing.

Thanks.

Re: symantec Brightmail gateway- SBG field extraction

mshakeb — Thu, 13 Aug 2020 12:05:08 GMT

i cannot download the add-on

Re: symantec Brightmail gateway- SBG field extraction

thambisetty — Thu, 20 Aug 2020 20:19:20 GMT

The Add-on is available now https://splunkbase.splunk.com/app/5181/

Re: symantec Brightmail gateway- SBG field extraction

J9h0m0e8 — Sun, 20 Sep 2020 20:17:36 GMT

try, i used the logic:

Error in 'SearchParser': Missing a search command before '^'. Error at position '151' of search query 'search source="tcp:3514" index="prueba" sourcetype...{snipped} {errorcontext = (?P<time>[^ ]+)\s+(?P}'.

Re: symantec Brightmail gateway- SBG field extraction

mshakeb — Mon, 21 Sep 2020 01:54:57 GMT

Hi,

Try this add-on for symantec Messaging gateway

https://splunkbase.splunk.com/app/5215/