About psheck117

psheck117 · ‎01-27-2014

Yeah, I realized that after I committed my transform... reading rfc1945 has been enlightening to say the least. Here is a crack at a proper REGEX for scheme, I will comment and add the http_referer_uri_extension after testing. REGEX = (? [a-zA-Z+.-]+)://(? S[^/]+)((? /.[^?]+))?((? ?.*))? Ha! Looking at my regex makes me question if I can tighten it a little better too.

psheck117 · ‎01-27-2014

Here is the full regex for my http_referer extraction. If you do something like this you may be surprised with what shows up as a referrer scheme. REGEX = (? \w+)://(? \S[^/]+)((? /.[^?]+))?((? \?.*))? I could probably get into the depth of http_referer_uri_extension, but that is hit or miss, and right now I am not sure I need the detail. Though, thinking about it, I could slip it in there. My first inclination was to break it out into multiple extractions too.

psheck117 · ‎01-24-2014

Thanks Martin! I will check out & use SOURCE_KEY, I knew I was missing something. As for my regex, definitely not going to end on ://. Though, there is only one place in the event that will exist, http:// or https:// in the referrer field, if it exists at all. I didn't want to put my whole regex into the question, so left at the first extracted field. Thanks again!

psheck117 · ‎01-24-2014

I am working on some http_referer analysis from my proxy logs, seems like an interesting thing to do. I want to do an additional search time field extraction and rip apart the http_referer field to provide more search functionality from the data. Can I do something like: transforms.conf: REGEX = field=http_referrer ^(? \w+):// *Yes, I realize my field name isn't the same as the RFC... haha, official misspelling 😕 I can build the whole thing out with a single line, and I am sure the hardware can handle the overhead without issue (I hope), but I'd rather have field anchor of some sort to go off of. Am I missing something on this? After thoughts: I can do a content match on the :// as there is nothing in the logs that should contain that combination of characters in ASCII, any colons in the URI will be in hex or something else. Thanks.

psheck117 · ‎12-10-2013

Hi ITSD, Ran into a similar problem where the logs were coming in but the date timestamp of the logs were all off. I recommend reading the docs on props.conf and then explicitly setting the following things in your props.conf file: TIME_PREFIX TIME_FORMAT TZ By explicitly setting your TIME_FORMAT it won't guess and you'll be able to help it understand when it occurred. I hope that helps.

psheck117 · ‎12-10-2013

Hi ageld, as a recommendation for easier reading/use for your users you may want to use the CIM (Common information model) fields for your header. I've used various flavors of field extractions for Blue Coat proxy logs and it's really nice to have a sane and understandable means to search the logs. Another plus side, coming from a Security perspective is correlation of activity in the proxy to other sourcetypes. On the configuration of time, setting the Time Zone is helpful, we ran into the pitfall of having to explicitly set the pattern for matching the timestamp too. Happy hunting!

Posts	6
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎12-10-2013

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

Extracting fields from an existing Field

Re: Extracting fields from an existing Field

Re: Extracting fields from an existing Field

Re: Extracting fields from an existing Field

Extracting fields from an existing Field

Re: Bluecoat Events

Re: Could someone provide some tips and tricks to ...