Splunk Search

symantec Brightmail gateway- SBG field extraction

thambisetty
Super Champion

Hi,
I am getting the logs from SBG,but splunk couldnt able to index those logs. I need to index those logs. I did field extraction for first 3 fields are common in every event. The main problem is The next fields depend up on the 3rd field that is action. Now i want to extract those fields and i need to name it for search purpose. below are the example events.
1> jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

2> Jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e7-53b9ce61f951|IRCPTACTION|oceane-ias@orange.com|annotate

3>Jul 7 04:02:01 wipro-blr-out01 bmserver: 1404685921|cb5bdd57-f792c6d000001154-e7-53b9ce61f951|VERDICT|oceane-ias@orange.com|content_300|default|legal disclaimer

if u see above events fisrt 3 fields are common so i named it. Now i want name the rest of fields by using following search.

index=main sourcetype=ec_sbg_outbound action=accept----- now i will get all the events of action=accept it has only one field after action so i need to name for that field.
like that i have to do all the action types..
please help me
in advance thanks.....

————————————
If this helps, give a like below.
1 Solution

thambisetty
Super Champion

https://splunkbase.splunk.com/app/5181/

————————————
If this helps, give a like below.

View solution in original post

mshakeb
Loves-to-Learn Lots

Hi,

Try this add-on  for symantec Messaging gateway 

https://splunkbase.splunk.com/app/5215/

 

0 Karma

Richfez
SplunkTrust
SplunkTrust

Since you have the extraction for the first three fields, I suspect you can make the extraction for the remaining ones, you just need to know how to do that and make it work right.

In $splunkhome/etc/apps/myappname/local/transforms.conf (or wherever) you will need to create several REGEX statements. You have log lines like:

jul 7 04:02:01 wipro-blr-out01 ecelerity: 1404685921|cb5bdd57-f792c6d000001154-e6-53b9ce619b84|ACCEPT|203.91.221.85:50090

So, use your REGEX you have for the first three fields, only don't extract the third field yet. Instead, include from the third field to the end of the line as something like "sbg_extra_info". BTW, I assume there's some "header" type of information that's not really "field1" - i.e. timestamp and so on. It doesn't matter for my explanation, I just mention it to prevent confusion below.

Some "pseudo regex" - meaning you may have to escape pipes, and honestly I just whipped it up so it's probably totally wrong, but it's close enough for our purposes:

[sbg-message-parse]
REGEX = ^(?<timestamp>[^ ]*\s+[^ ]*\s+[^ ]*)\s+(?<host>[^ ]*)\s+(?<some_other_field>[^ :]*)[:]\s+(?<field1>[^|]*)|(?<field2>[^ ]*)\s+(?<sbg_extra_info>.*)

But the important part was field1, field 2, then "everything else" as sbg_extra_info.

Now, also in that same transforms, create more stanzas, one for each of the type of service (ACCEPT, IRCPTACTION, VERDICT, etc...). Use "SOURCE_KEY = sbg_extra_info" to start by using that field for this extraction.

[sbg-extrainfo-accept-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>ACCEPT)|(?<accept_field1>[^|]*)(?<accept_field2>...

[sbg-extrainfo-ircptaction-parse]
SOURCE_KEY = sbg_extra_info
REGEX = (?<service>IRCPTACTION)|(?<ircptaction_field1>[^|]*)(?<ircptaction_field2>...

Notice in each of those, I pull out the "service" (IRCPTACTION, ACCEPT... ) as well, then the rest of the REGEX just extracts whatever appropriate for the rest of the message. Add more fied extractions and stanzas as required.

Lastly, you have to call of these from props.conf. Order is important in that you have to pull out your sbg_extra_info FIRST. All the rest are on equal footing because there's no "nested" dependencies, just that one field needs to be created first. So, in $splunkhome/etc/apps/myappname/local/props.conf, call them all.

[mysourcetype]
REPORT-sbg_info = sbg-message-parse,sbg-extrainfo-accept-parse,sbg-extrainfo-ircptaction-parse,...

That should be it. I usually recommend getting the main sbg-message-parse right first, then proceeding with the rest. That way you can tweak each regex as a rex in a search directly and get it just right before committing it to your transforms.conf file.

sorenmaigaard
Path Finder

Hi

Did you ever find a way to parse these logs?

Best
Soren

thambisetty
Super Champion

Yes, I have created a TA which extracts only required event types.

————————————
If this helps, give a like below.

to4kawa
SplunkTrust
SplunkTrust

alexling75
Explorer

Hi thambisetty,

 

Is it possible to share this TA ?

 

Thanks in advance.

 

Best regards,

Alex Ling

thambisetty
Super Champion

how can I share?

I will try to upload to Splunkbase.

————————————
If this helps, give a like below.

thambisetty
Super Champion

https://splunkbase.splunk.com/app/5181/

————————————
If this helps, give a like below.

View solution in original post

thambisetty
Super Champion

The Add-on is available now https://splunkbase.splunk.com/app/5181/

————————————
If this helps, give a like below.

mshakeb
Loves-to-Learn Lots

i cannot download the add-on

0 Karma

alexling75
Explorer

Hi,

 

I am unable to see the page in splunkbase.

Can assist?

Thanks.

AL

thambisetty
Super Champion

@alexling75 

I have emailed you.

the one I uploaded to splunkbase is still pending for Splunkbase engineer review.

————————————
If this helps, give a like below.

alexling75
Explorer

Hi Balaji,

I am grateful for your kind sharing.

Thanks.

AL

alexling75
Explorer

Hi thambisetty,

Can send to my email cwlingatyahoo.com?

Thanks.

Alex Ling

 

 

alecdhuse
Engager

I created this regex extraction, that extracts fields for the majority of Symantec Messaging Gateway's logs:

^<142>(?P<date>\w+\s+\d+)\s+(?P<time>[^ ]+)\s+(?P<server>\w+)\s+(?P<process_name>[a-z]+)\[(?P<process_number>\d+)[^ \n]* (?P<process_id>[^\|]+)\|(?P<message_id>[^\|]+)\|(?P<action>IRCPTACTION|VERDICT|UNTESTED|FIRED|SENDER|LOGICAL_IP|EHLO|MSG_SIZE|MSGID|SOURCE|SUBJECT|ORCPTS|TRACKERID|ATTACH|UNSCANNABLE|VIRUS|DELIVER|ACCEPT)(?:(?:(?<=ACCEPT|DELIVER|LOGICAL_IP)\|(?P<src>[^:\s]+)(?::(?P<port>[0-9]+))?(?:\|(?P<to>[^\s]+))?)|(?:(?<=FIRED|IRCPTACTION|ORCPTS|TRACKERID|UNTESTED|VERDICT)\|(?P<recipient>[^\s\|]+)(?:\|)?(?P<result>[a-z][^\|\s]+)?(?:\|(?P<result_2>[a-z][^\|]+))?(?:\|(?P<result_3>.+))?)|(?:(?<=SENDER)\|(?P<from>[^\s]+))|(?:(?<=MSG_SIZE)\|(?P<msg_size>\w+))|(?:(?<=SUBJECT)\|(?P<subject>.*))|(?:(?<=ATTACH)\|(?P<attachment>.+))|(?:(?<=UNSCANNABLE)\|(?P<reason>.+))|(?:(?<=VIRUS)\|(?P<virus_name>.+))|(?:(?<=EHLO)\|(?P<fqdn>.+)))?

I wrote a short blog post about it here: http://alec.dhuse.com/?p=217

J9h0m0e8
Engager

try, i used the logic:  

Error in 'SearchParser': Missing a search command before '^'. Error at position '151' of search query 'search source="tcp:3514" index="prueba" sourcetype...{snipped} {errorcontext = (?P<time>[^ ]+)\s+(?P}'.
 
 
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!