About ccsfdave

ccsfdave · ‎05-20-2016

I set the darn thing over 3 years ago and it is not any of my usual passwords. Is there a way to recover the REST API password? I am talking about the one that defaults as "changeme" Also if I can recover it, will anything break e.g. forwarders? Thanks!

ccsfdave · ‎05-18-2016

I may have to create 100s over time but for now, just one or two 😉

ccsfdave · ‎05-18-2016

Greetings, I have read through the Knowledge Manager Manual on summary indexes, but am left with a question for my usecase. Our environment aggregates the internet connection for many departments into one pipe out. The Departmental users want to review these logs, but I don't want them seeing outside the walls of that department. We segment our internal network by IP range, so Dept A would have an IP of 10.1.0.0 and Dept B has 10.2.0.0 etc. So I would like to take our index=pan_logs src_ip=10.1.* and send that to a summary index of pan_dept_a. I would then grant the deptA group access to that summary index and that is how they could search for Palo Alto firewall data w/o accessing Dept B traffic logs on the pan_logs index. Does this make sense and more importantly, conceptually, will it work? Thanks!

ccsfdave · ‎05-18-2016

Hi @mokuso I have replicated all the dashboards in splunk except for the ones that extract the OS and Application. I just couldn't get the regex right. But those two are nice to haves. Anyway, I passed on your question for the next release to the lead on PVS over here and he seems happy with what we have but if you would like to have a more open conversation or like us to beta test, drop me a line at david (dot) geller (at) sfgov (dot) org. OH BTW, on the "replicated" dashboards, I added host, sourcetype (internal or external) and a time picker. So those are improvements on the PVS canned dashes as well. Thanks, Dave

ccsfdave · ‎05-16-2016

\{0d}\{0a}.*?\{0d}\{0a}.*?\{20}(?<fields>[^\{]+)\{0d}\{0a} Is what I used in this case but I have found what I am after shows up in different places depending on the application so building a field is just not going to work for me...I'll keep thinking about it or maybe abandon it. Thanks!

ccsfdave · ‎05-16-2016

oh my regex in regex101.com just goes until the last {0d} in the _raw. I need it to stop at the next {0d}. BTW, how does {0d}{0a} skip the first instance of "{0d}{0a}" and start matching at the second?

ccsfdave · ‎05-16-2016

@richgalloway Thanks for the regex! See my addendum above

ccsfdave · ‎05-16-2016

@sundareshr Thanks for the regex but I guess what I am really after is sort of counting {00}, {0d}{0a}, and {20} to find the right location. Your regex is dependent on "Server" being in there but when the app is Oracle Java SE, "Server" will not precede the desired extraction. I do use regex101 but probably need a book or better tutorial because I can mash on it for a while and not make progress. My regex was: {0d}{0a}.+:{20}(? .+){0d} But what I really need is the 2nd occurrence of the "{0d}{0a}" and then I need it to stop on time...the collection seems to go on for too far

ccsfdave · ‎05-16-2016

Here is the data I am trying to parse. I actually want to extract a number of fields but cannot figure out how to parse through the {0d}{0a}{20}s. For this question, what regex will pull out "Microsoft-IIS/8.5"? May 16 06:56:02 75-vw-win7ns.net-10-1-3.dhcp.company.org pvs: 10.x.x.x:46168|10.y.y.y:80|6|6852|HTTP 4xx Detection (Client)|4{00}E{00}2{00}6{00}8{00}C{00}6{00}F{00}<{00}/{00}P{00}r{00}o{00}p{00}e{00}r{00}t{00}y{00}>{00}<{00}/{00}H{00}o{00}o{00}k{00}2{00}>{00}<{00}/{00}H{00}o{00}o{00}k{00}s{00}>{00}<{00}P{00}a{00}y{00}l{00}o{00}a{00}d{00}{20}{00}T{00}y{00}p{00}e{00}={00}"{00}i{00}n{00}l{00}i{00}n{00}e{00}"{00}/{00}>{00}<{00}T{00}a{00}r{00}g{00}e{00}t{00}H{00}o{00}s{00}t{00}>{00}D{00}T{00}-{00}S{00}C{00}C{00}M{00}P{00}R{00}O{00}D{00}0{00}1{00}.{00}A{00}D{00}.{00}S{00}F{00}G{00}|HTTP/1.1{20}401{20}Unauthorized{0d}{0a}Content-Type:{20}text/html{0d}{0a}Server:{20}Microsoft-IIS/8.5{0d}{0a}WWW-Authenticate:{20}Negotiate{0d}{0a}WWW-Authenticate:{20}NTLM{0d}{0a}X-Powered-By:{20}ASP.NET{0d}{0a}Date:{20}Mon,{20}16{20}May{20}2016{20}13:55:59{20}GMT{0d}{0a}Content-Length:{20}1293{0d}{0a}{0d}{0a}<!DOCTYPE{20}html{20}PUBLIC{20}"-//W3C//DTD{20}XHTML{20}1.0{20}Strict//EN"{20}"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">{0d}{0a}<html{20|NONE May 16 06:56:01 75-vw-win7ns.net-10-1-3.dhcp.company.org pvs: 10.x.x.x:46168|10.y.y.y:80|6|6852|HTTP 4xx Detection (Client)|--aAbBcCdDv1234567890VxXyYzZ{0d}{0a}content-type:{20}text/plain;{20}charset=UTF-16{0d}{0a}{0d}{0a}{ff}{fe}<{00}M{00}s{00}g{00}{20}{00}S{00}c{00}h{00}e{00}m{00}a{00}V{00}e{00}r{00}s{00}i{00}o{00}n{00}={00}"{00}1{00}.{00}1{00}"{00}>{00}{0d}{00}{0a}{00}{09}{00}<{00}I{00}D{00}>{00}{{00}4{00}2{00}7{00}D{00}C{00}1{00}C{00}C{00}-{00}1{00}0{00}7{00}4{00}-{00}4{00}7{00}C{00}3{00}-{00}9{00}0{00}0{00}6{00}-{00}D{00}0{00}4{00}7{00}8{00}C{00}3{00}E{00}E{00}0{00}6{00}|HTTP/1.1{20}401{20}Unauthorized{0d}{0a}Content-Type:{20}text/html{0d}{0a}Server:{20}Microsoft-IIS/8.5{0d}{0a}WWW-Authenticate:{20}Negotiate{0d}{0a}WWW-Authenticate:{20}NTLM{0d}{0a}X-Powered-By:{20}ASP.NET{0d}{0a}Date:{20}Mon,{20}16{20}May{20}2016{20}13:55:59{20}GMT{0d}{0a}Content-Length:{20}1293{0d}{0a}{0d}{0a}<!DOCTYPE{20}html{20}PUBLIC{20}"-//W3C//DTD{20}XHTML{20}1.0{20}Strict//EN"{20}"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">{0d}{0a}<html{20|NONE

ccsfdave · ‎05-15-2016

I think you got it mokuso. I did some testing with removing variables from the sort and am now sure I understand how it is working. Though the Infos and Nones dwarft the rest of my stats, I could choose to remove those from earlier in the search. Anyway, thanks so much! Dave

ccsfdave · ‎05-13-2016

Greetings, I am trying to replicate the dashboards found in the Tenable PVS environment. First, this is the dashboard I am after: Note the IPs are top 10 and the colors are the severity. From the data, I have this chart which I think gets me close, but for it to work, I would have to sort by Critical, then by High, then Medium, etc and then take the top 10 IP addresses. index=pvs | chart count(eval(PVS_risk="CRITICAL")) AS CRITICAL , count(eval(PVS_risk="HIGH")) AS HIGH, count(eval(PVS_risk="MEDIUM")) AS MEDIUM, count(eval(PVS_risk="LOW")) AS LOW, count(eval(PVS_risk="INFO")) AS INFO, count(eval(PVS_risk="NONE")) AS NONE by src Can anyone offer any pointers or similar dashboards I may be able to leverage? BTW, I have the PVS app configured and all the dashes displaying, but I wanted to get ALL of the PVS dashboards into Splunk. Thanks!

ccsfdave · ‎05-13-2016

Well, I can't say I have learned anything from this but what I did is just do a field extraction through the GUI and pasted: (?P<PVS>pvs): (?P<src>[^:]+):(?P<src_port>\d{1,5})\|(?P<dest>[^:]+):(?P<dest_port>\d{1,5})\|(?P<protocol>\d{1,3})\|(?P<PVS_pluginid>\d{1,5})\|(?P<PVS_eventname>[^\|]+)\|(?P<PVS_data>[^\|]+)\|(?P<PVS_data2>[^\|]+)?\|(?P<PVS_risk>[^\|]+) Into the regex window. It is now working. Does anyone know whether the field extractor changes the props.conf on the deployment-apps or apps directory?

ccsfdave · ‎05-13-2016

@rich7177 thanks for the response Unfortunately the linemerge didn't have any affect. I put a $ in the regex tester: https://regex101.com/#pcre and it did nothing. What does sorta work is put an new line (\n) just before the last parenthesis however it reads everything except the last entry. Which may be ok as logs roll in but it is not tested. I suppose I will.

ccsfdave · ‎05-13-2016

May 13 09:15:44 10.x.x.x pvs: 10.x.x.x:53|10.y.y.y:53|17|7024|DNS Client Queries|PVS has observed this host perform a DNS lookup. The most recent DNS query performed was for: |hostname.some.ORG to the server at 10.y.y.y|NONE May 13 09:15:42 10.x.x.x pvs: 10.y.y.y32:61565|10.z.z.z:80|6|7041|HTTP request detection|The following GET/POST request was observed:|DIP: 10.z.z.z:80;URI: /department/Admin/ProjectManage/Lists/Develop%20New%20PM%20tracking/Research;Referer: None;Host: hostname2.some.org;Query: YES;PROTO: 1.0|NONE May 13 09:15:42 10.x.x.x pvs: 10.y.y.y32:61566|10.z.z.z:80|6|7041|HTTP request detection|The following GET/POST request was observed:|DIP: 10.z.z.z:80;URI: /committee_name/committees/budget_perf/DeptITPlansFY201516ThroughFY201920/CON;Referer: None;Host: hostname2.some.org;Query: NO;PROTO: 1.0|NONE May 13 09:15:42 10.x.x.x pvs: 10.y.y.y32:61565|10.z.z.z:80|6|7041|HTTP request detection|The following GET/POST request was observed:|DIP: 10.z.z.z:80;URI: /department/Admin/ProjectManage/Lists/Develop%20New%20PM%20tracking/Research;Referer: None;Host: hostname2.some.org;Query: NO;PROTO: 1.0|NONE

ccsfdave · ‎05-13-2016

Greetings, I have a few PVS's coming through syslog via TCP. I have set index=pvs, sourcetype=pvs:internal (for these, there will be "externals" coming down the pipe in a few weeks) and the host=. I have attempted to comment out the syslog stanza of the props.conf and collapsed the extract into the local/props.conf stanza automagically created when I set the sourcetype on the heavy forwarder to [pvs:internal]. So I now have this in my /opt/splunk/etc/apps/pvs/local/props.conf: [pvs:internal] DATETIME_CONFIG = NO_BINARY_CHECK = true category = Custom disabled = false pulldown_type = true TRANSFORMS-changesourcetype = set_sourcetype_pvs EXTRACT-PVS,src,src_port,dest,dest_port,protocol,PVS_pluginid,PVS_eventname,PVS_data,PVS_data2,PVS_risk = (?P<PVS>pvs): (?P<src>[^:]+):(?P<src_port>\d{1,5})\|(?P<dest>[^:]+):(?P<dest_port>\d{1,5})\|(?P<protocol>\d{1,3})\|(?P<PVS_pluginid>\d{1,5})\|(?P<PVS_eventname>[^\|]+)\|(?P<PVS_data>[^\|]+)\|(?P<PVS_data2>[^\|]+)?\|(?P<PVS_risk>[^\|]+) When I went through the regex of the extract into https://regex101.com/ it seems to grab every other event (which may be a different issue), but I wanted to verify the regex. Anyway, I am not getting any extractions which is my real issue. Can anyone offer suggestions? Thanks, Dave

ccsfdave · ‎04-29-2016

Should I use standard or CEF for sending PVS data to my syslog?

ccsfdave · ‎04-26-2016

Well, the above worked for me. In our case we have 675GB SSD RAID 1 each on two indexers and they were full with the default settings. I finally got the 150TB of spinning drives mounted in as cold but nothing was rolling over to it. So I did a search of my data to see how far it went back. Not sure this was scientific in anyway but I decided to 1/3 the default settings above with the end results being . The end result was I brought my hot drives to 60% and 72% utilization. So we will go forward with this config until I get more hot drives.

ccsfdave · ‎04-25-2016

I changed the coldPath on all my indexes to volume:cold I created a /opt/splunk/etc/system/local/indexes.conf on my SH and Indexers maxWarmDBCount = 50 maxHotSpanSecs = 2592000 Anything else I should or shouldn't have done?

ccsfdave · ‎04-25-2016

Hmm, sorry it's not working for you... I stripped it down more and the eval does work for me (obviously). Not sure why. | convert dur2sec(duration) | stats sum(duration) as sumdur |eval "Time Connected"=tostring(sumdur, "duration") duration is a field in my data. ignoring the bucket of one day tied to the _time (parses the search by date)...I just ran my new search for 24h which is the same thing (and much quicker). The above results in: sumdur Time Connected 106846 1+05:40:46 Yeah...I live on VPN...

ccsfdave · ‎04-25-2016

I have a search to find VPN connection durations, which I built a long time ago and probably with the help of answers.splunk.com. But here are the relevant parts if you can pick it apart for your usecase: | convert dur2sec(duration) |bucket _time span=1d | stats sum(duration) as sumdur by _time src_ip |eval "Time Connected"=tostring(sumdur, "duration") |fields - sumdur |rename _time as Date | convert timeformat=%m/%d/%Y ctime(Date)

ccsfdave · ‎04-25-2016

So, I got the 150TB cold, but they are mounted into /mnt/splunk1/cold and /mnt/splunk2/cold. I figured that may cause issues with the indexers, so I made symlinks to /opt/splunk/var/lib/splunk/cold on each of the indexers to prevent issues with which indexer Splunk wants to write to. I am now thinking about changing the indexes.conf and adding to the volume stanza: # One Volume for Cold [volume:cold] path = /opt/splunk/var/lib/splunk/cold # 150000GB (150TB) maxVolumeDataSizeMB = 150000000 Then changing the cold locations from: coldPath = volume:primary/defaultdb/colddb to coldPath = volume:cold/defaultdb/colddb The ES definitions are: coldPath = $SPLUNK_DB/audit_summarydb/colddb I would like to change that too, similar to above: coldPath = volume:cold/audit_summarydb/colddb Thoughts? Guidance?

ccsfdave · ‎04-20-2016

Yup you got it! | rex field=url "(?<domain>\w+\.\w+)\/"

ccsfdave · ‎04-20-2016

@somesoni2 You have it, but help me understand it so that I may apply it to my search. As @Rhin0Crash stated the Palo Altos see the field as "url" so my base search is: index=pan_logs sourcetype=pan* src_ip=x.x.x.x url=*

ccsfdave · ‎04-20-2016

I have been through the field extractor, answers.splunk.com, and the interwebs looking for help on this one. So our Palo Alto will give us the URLs of sites visited - here is a sample: crl.microsoft.com/pki/crl/products/MicRooCerAut2011_2011_03_22.crl safebrowsing-cache.google.com/ p4-a2lp5grl52xoy-qpo2s4ky6vs36rpb-794312-s1-v6exp3-v4.metric.gstatic.com/ de.tynt.com/deb/v2?id=dZxfWCGner46jsacwqm_6l&r=lyricstranslate.com/en/l039amour-c039est-pour-rien-love-nothing.html a248.e.akamai.net/ I would like to be able to extract the domains e.g. microsoft or microsoft.com google or google.com gstatic or gstatic.com tynt or tynt.com akamai or akamai.net I would think that the way to go about it is to look for the FIRST .com, .net, .org etc and then work back to the previous . to grab the domain but that is beyond me. Can anyone help?

ccsfdave · ‎03-08-2016

shoot, in /opt/splunk/etc/apps/sf_syslog_inputs/local/inputs.conf I had: [monitor:///var/log/syslog/ngf0*/*.log] index = pan_logs sourcetype = pan no_appending_timestamp = true host_segment = 4 Which I have now changed to pan_logs and bounced the Fwdr. Let's see what happens

Posts	216
Solutions	20
Karma Given	44
Karma Received	25
Member Since	‎03-06-2012

Online Status	Offline
Date Last Visited	‎04-19-2024 05:10 AM

Tanium app inputs

Update the example data?

Field Extracts not working for Darktrace Connector

Why is the IMAPMailbox Mail not indexing?

Cisco eStreamer eNcore Add-on for Splunk: Why am I...

Distributed Management Console not visible at all.

How do you draw lines on a map

How do you place icons on a map

Why am I missing fields in search results running ...

How do you compare a Single Value visualization to...

REST API password

Re: Is it possible to assign roles to only have ac...

Is it possible to assign roles to only have access...

Re: How to replicate dashboards in the Tenable Net...

Re: How do I write the regex to extract this field...

Re: How do I write the regex to extract this field...

Re: How do I write the regex to extract this field...

Re: How do I write the regex to extract this field...

How do I write the regex to extract this field fro...

Re: How to replicate dashboards in the Tenable Net...

How to replicate dashboards in the Tenable Network...

Re: Tenable Network Security PVS App for Splunk: H...

Re: Tenable Network Security PVS App for Splunk: H...

Re: Tenable Network Security PVS App for Splunk: H...

Tenable Network Security PVS App for Splunk: How t...

Tenable Network Security PVS App for Splunk: Shoul...

Re: Can anyone provide guidance on my plan to conf...

Re: Can anyone provide guidance on my plan to conf...

Re: How to search duration using two timestamps?

Re: How to search duration using two timestamps?

Can anyone provide guidance on my plan to configur...

Re: How to write the regex to extract the domains ...

Re: How to write the regex to extract the domains ...

How to write the regex to extract the domains from...

Re: Palo Alto app not parsing the sourcetype