Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About daniel_augustyn
daniel_augustyn
Contributor
Member since:
09-20-2015
02-15-2021
Community Statistics
| Posts | 115 |
| Solutions | 6 |
| Karma Given | 23 |
| Karma Received | 9 |
| Member Since | 09-20-2015 |
Activity Feed
- Got Karma for Re: How to get rid of warning sign "search is waiting for input ..." and display the data in dashboard panel?. 09-08-2021 12:09 PM
- Karma Re: Is there an add-on to monitor and parse DNS logs from Windows 2012 R2 DNS servers? for mosman_splunk. 06-05-2020 12:48 AM
- Karma Re: Dashboard Statistics Table Not Showing for woodcock. 06-05-2020 12:48 AM
- Karma Re: Splunk Enterprise Security setup: Why am I seeing error "unable to distribute to peer...because replication was unsuccessful."? for Masa. 06-05-2020 12:48 AM
- Karma Re: How to pull logs into Splunk from Proofpoint via APIs or any other methods from a Proofpoint Cloud instance? for eckolp2003. 06-05-2020 12:48 AM
- Karma Re: How to pull logs into Splunk from Proofpoint via APIs or any other methods from a Proofpoint Cloud instance? for ChrisBell04. 06-05-2020 12:48 AM
- Karma Re: How to ingest logs from a single file and split them into 2 source types or indexes? for sundareshr. 06-05-2020 12:48 AM
- Karma Re: Filtering Unwanted Events for jconger. 06-05-2020 12:48 AM
- Karma Re: Is there an add-on to monitor and parse DNS logs from Windows 2012 R2 DNS servers? for spayneort. 06-05-2020 12:48 AM
- Karma Re: After dragging a dashboard panel to expand it, how can I keep it this size, even after refreshing the page? for mdorobek. 06-05-2020 12:48 AM
- Karma Re: How to configure Splunk to extract the correct year from the date stamp in my sample log? for ryanoconnor. 06-05-2020 12:48 AM
- Karma Re: Splunk Add-on for Check Point OPSEC LEA: After upgrading to Splunk Enterprise 6.4, why does the Manage Connections page keep loading? for apilger_splunk. 06-05-2020 12:48 AM
- Karma Re: Splunk Add-on for Check Point OPSEC LEA: After upgrading to Splunk Enterprise 6.4, why does the Manage Connections page keep loading? for hjauch_splunk. 06-05-2020 12:48 AM
- Karma Re: Splunk Add-on for Check Point OPSEC LEA: After upgrading to Splunk Enterprise 6.4, why does the Manage Connections page keep loading? for ppablo. 06-05-2020 12:48 AM
- Got Karma for Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE". 06-05-2020 12:48 AM
- Got Karma for Troubleshooting Data Model Network_Traffic. 06-05-2020 12:48 AM
- Got Karma for Re: How to reset a license warning to avoid a license violation?. 06-05-2020 12:48 AM
- Got Karma for Active Directory DNS logs - extract domain name without periods. 06-05-2020 12:48 AM
- Got Karma for How to pull logs into Splunk from Proofpoint via APIs or any other methods from a Proofpoint Cloud instance?. 06-05-2020 12:48 AM
- Got Karma for Splunk Enterprise Security setup: Why am I seeing error "unable to distribute to peer...because replication was unsuccessful."?. 06-05-2020 12:48 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 1 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 0 |
01-16-2016
12:53 PM
I can't find how to extract the User Agent field from the Blue Coat proxy logs. I couldn't find the correct answer yet on the forum. All of the answers I went through had regex that didn't work correctly.
REGEX = (?<date>[^\s]+)\s+(?<time>[^\s]+)\s+(?<time_taken>[^\s]+)\s+(?<c_ip>[^\s]+)\s+(?<cs_username>[^\s]+)\s+(?<cs_auth_group>[^\s]+)\s+(?<x_exception_id>[^\s]+)\s+(?<filter_result>[^\s]+)\s+\"(?<category>[^\"]+)\"\s+(?<http_referrer>[^\s]+)\s+(?<sc_status>[^\s]+)\s+(?<action>[^\s]+)\s+(?<cs_method>[^\s]+)\s+(?<http_content_type>[^\s]+)\s+(?<cs_uri_scheme>[^\s]+)\s+(?<cs_host>[^\s]+)\s+(?<cs_uri_port>[^\s]+)\s+(?<cs_uri_path>[^\s]+)\s+(?<cs_uri_query>[^\s]+)\s+(?<cs_uri_extension>[^\s]+)\s+[\"]{0,1}(?<http_user_agent>[^\"]+)[\"]{0,1}
[\"]{0,1}(?<http_user_agent>[^\"]+)[\"]{0,1}
Extracts 'dashes like this, together with a dvc_ip.
10.106.4.11
10.106.4.11
Does anyone have this issues sorted out already?
... View more
- Tags:
- splunk-enterprise
01-16-2016
12:46 PM
None of these regex work to extract http_user_agent. Did anyone get it right?
... View more
01-16-2016
07:57 AM
Haha, I did that already. I had to adjust a little bit the bluecoat add-on transform.conf file, two last fields in Regex were messing the logs and I had to adjust them a bit to include dashes while extracting. It was mostly UA field that in Regex that was really messing the logic and didn't want to extract the fields from the logs since it didn't recognize a dash if the UA was missing in the specific log.
... View more
01-15-2016
11:06 AM
thanks for the input
... View more
01-15-2016
10:02 AM
Not sure why regex is getting cut off. I attached the regex below.
... View more
01-15-2016
09:42 AM
Under the "Extract Fields" page, it shows that these fields should have been extracted. All fields are recognized correctly and circled with different colors.
( ?=[^p]*(?:portal.threatpulse.net|p.*portal.threatpulse.net))^(?P[^ ]+)\s+(?P[^ ]+)[^ \n]* (?P\d+)(?:[^ \n]* ){9}(?P\w+)[^"\n]*"(?P[^"]+)"\s+\-\s+(?P\d+)[^ \n]* (?P[^ ]+)\s+(?P\w+)\s+(?P[^ ]+)[^ \n]* (?P[a-z]+)\s+(?P\w+\.\w+\.\w+)\s+(?P[^ ]+)(?:[^ \n]* ){5}(?P\d+\.\d+\.\d+\.\d+)\s+(?P\d+)\s+(?P\d+\s+\-)
Here is the log which fields are not getting extracted:
2016-01-15 17:08:58 56 10.167.7.93 - - portal.domain.net x.x.x.x None - - OBSERVED "Technology/Internet" - 302 TCP_NC_MISS GET text/html http portal.threatpulse.net 80 / - - - 172.16.167.104 177 80 - "none" "none" 3 454f7877563349f7-00000000027dbbf5-00000000569927aa
Well, only half of fields are getting extracted to be more specific, protocol, dest_host, and dest_port are not extracted. Not sure why this is not working as any others. It should have been.
... View more
01-14-2016
05:13 PM
I just did a regex for proxy fields extractions and it seems that is not working as it should have. Not sure why. Fields for some of the proxy logs are getting extracted but some don't. The weird thing about this is that the regex works fine on regex101.com and additionally, when I try to redo the rex using drop down under an event for "Extract Fields", I am getting redirected to the "extract field" page and it shows in colors that these fields should have been extracted. Not sure why they are not getting extracted. Any thoughts?
... View more
01-14-2016
11:56 AM
Thanks for moving it.
... View more
01-14-2016
11:55 AM
I tried that too, and it wasn't really helpful. I will need to dump all of the logs that the BC add-on doesn't recognize and sort them and see how many rex queries I will need to write. Thanks for your help!
... View more
01-13-2016
05:42 PM
If I would do one long rex for each different type of proxy logs, there would be a lot of work to do. Since you have one long rex which is really specific to that single type of proxy logs, you would need to create tons of them. I was looking for something more general, and wasn't sure how to create it. In my logs, I've been seeing a lot of different field structures. Not sure why.
2016-01-14 00:42:32 284 10.130.16.102 - - proxy.domain.net x.x.x.x None - - PROXIED "Social Networking" http://www.domain.com/article/david-bowie-blackstar-album-sales-networth 200 TCP_NC_MISS GET application/javascript;%20charset=utf-8 http platform.domain.com 80 /widgets.js - js "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" 172.16.130.10 27626 726 - "Widgets" - none
OR
2016-01-14 00:42:03 1 10.130.0.156 - - 0.0.0.0 - Invalid - invalid_request PROXIED - - 400 TCP_NC_MISS unknown - - - 0 / - - - 172.16.130.10 842 152 - "none" "none" none
I have plenty of others that don't lay out together. I would work on it and try to figure something out, but if you have any idea how to approach it, I would really appreciate.
... View more
01-13-2016
03:50 PM
What is the fastest way to extract fields from Blue Coat proxies logs in Splunk?
Is it better doing a simple Regex query for a single field like:
^(?:[^,\n]*,){5}(?P<hostname>[^\.]+)
or is it better doing a longer Regex and be more specific while extracting fields, something like:
^(?P<date>\d+\-\d+\-\d+)[^ \n]* (?P<time>\d+:\d+:\d+)\s+(?P<time_taken>[^ ]+)(?:[^\.\n]*\.){3}\d+\s+\-\s+\-\s+\-\s+(?P<filter_result>\w+)[^"\n]*"(?P<category>[^"]+)[^"\n]*"\s+(?P<http_referrer>[^ ]+)\s+(?P<http_response>[^ ]+)\s+(?P<action>[^ ]+)\s+(?P<cs_method>\w+)[^ \n]* (?P<http_content_type>\w+/\w+)[^ \n]* (?P<protocol>\w+)\s+(?P<dest_host>[^ ]+)[^ \n]* (?P<dest_port>\d+)\s+(?P<cs_uri_path>[^ ]+)[^\-\n]*\-\s+(?P<cs_uri_extension>\w+)\s+\-\s+(?P<dvc_ip>[^ ]+)[^ \n]* (?P<bytes_in>\d+)[^ \n]* (?P<bytes_out>\d+)[^\-\n]*\-\s+"(?P<http_content>[^"]+)
I assume doing a longer Regex would take a really long time, since these are really specific queries. Any other idea how to extract fields from Blue Coat proxies logs? I am addressing this question for the logs which Blue Coat add-on didn't extract.
... View more
01-04-2016
08:44 PM
The issue was because I added to much to my stanza for monitoring too many files at once and Splunk basically filled up the buffer. Since proxy logs are in gz format, it took long time for Splunk to catch up. Splunk will need to finish a single gz file before it could move to the next one. I also increased thruput in limints.conf.
... View more
01-04-2016
01:07 PM
I've been sending proxy logs to the FTP server and from there I installed an universal forwarder to send the logs to the Splunk indexers. They are all in a gz format. Everything was working fine until a day when I've noticed that proxy logs stopped getting indexed. There are about 10 subfolders and only 2 of them are still getting indexed, and the rest of the proxy logs had stopped getting indexed on the same day. How should I troubleshoot this?
Not sure why some of the subfolders with gz files (proxy logs from each site) has stopped getting indexed and the rest is still going.
... View more
01-01-2016
10:19 AM
I am going slightly over my license limit from time to time because of the Checkpoint firewall logs. Is there a way to aggregate some of the firewalls logs before start indexing them into the Splunk indexers? Or the only option would be to add another 20GB of license to Splunk.
... View more
- Tags:
- splunk-enterprise
12-21-2015
05:44 PM
Is there a way in Splunk to tag some specific logs and keep them for longer retention time? So for example, I want to tag several logs from firewall index, and I don't want these logs to be overwritten ever with new logs. How should I preserve these logs in Splunk, for example logs related to incidents, etc. Just the thoughts if Splunk has this feature?
... View more
12-15-2015
08:48 PM
Where should I deploy the Blue Coat Add-on for proxy SG logs? I'm running a Splunk indexer cluster with a couple of indexers, a master, and a search head. I wanted to find out where to install the app for the field extractions. Should this be done on the indexers? What about the add-on for Blue Coat, should this be installed on the search head and available for end users? I'm kind of confused how this should be deployed. Right now, I am pushing proxy logs from the FTP server to both indexers.
... View more
12-13-2015
04:33 PM
I can't find gzip2 file in the bin folder.
... View more
12-13-2015
04:30 PM
I can't find bzip2 in the bin directory, is there a way to threat done like gz files.
... View more
12-13-2015
04:04 PM
Would you mind sharing stanza for it?
... View more
12-13-2015
03:47 PM
Can you let me know what the stanza should be?
... View more
12-13-2015
03:05 PM
would that work on the Windows box?
... View more
12-13-2015
02:58 PM
no errors, would this be something related to https://answers.splunk.com/answers/8521/can-splunk-read-bluecoat-logs-formatted-and-compressed-as-log-gz-done-file-type.html
... View more
12-13-2015
02:57 PM
I am just fine with reading .gz files, I can't read gz.done files from the same folder.
... View more
12-13-2015
02:37 PM
And it doesn't collect these files.
... View more
12-13-2015
02:37 PM
That's what I have:
[monitor://E:\Server1\BCT-GW-SG\*.done]
sourcetype = bluecoat:proxysg:access:file
disabled = false
index=proxy
... View more
- « Previous
- Next »
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
02-15-2021
09:04 PM
|
Karma given to
