About bkirk

bkirk · ‎02-15-2019

This might not be the right place for this question but I see DNS request that seem to have a recordtype = ZERO in my splunk logs. I don't know what would be causing this, it doesn't seem like a valid type that DNS would even let you query for. 2/13/2019 11:48:59 AM 14E8 PACKET 000000A722BE61B0 UDP Rcv 10.10.10.106 d751 Q [0001 D NOERROR] ZERO www.microsoft.com I am seeing tons of these from different client on different servers for different queries. Doesn't seem like they get passed on to the downstream DNS for resolution, but I don't know if this ZERO record is a hiccup on the DNS servers or with clients there are times when see thousands of them and normally it is like a few hundred. Anyone else have Microsoft DNS logs and see random queries for record type ZERO? Thank you, Brian Kirk

bkirk · ‎01-20-2019

I installed your ta, added the script and the data is in splunk now but not sure it is going in correctly I get no data how does it know the right eventtype? I did change the folder to /var/log/splunk/hdhomerun hope that didn’t cause me the issues. Then I added the data input for the folder and made the sourertype=hdhomerun and the index=hdhomerun Is there something else I need to get the eventtype to work? Thank you.

bkirk · ‎11-14-2018

I think you want to add the following to your SPL: | streamstats current=f last(count) as last_count |eval Percent=if(isnotnull(last_count),round((count-last_count)/last_count*100,2)+100,100) If that isn't what you want at least the streamstats should get you want you want. and then it is just about doing your calculation. Thank you, Brian

bkirk · ‎11-14-2018

I think we had this same problem. Please check this thread out it seemed to help us: https://answers.splunk.com/answers/151428/how-to-prevent-internal-splunk-audit-logs-from-being-written-to-var-log-audit.html

bkirk · ‎11-13-2018

I have remembered a lot of things that I forgot 🙂 Here is a good answer on the format function does a good job explaining it: https://answers.splunk.com/answers/351834/how-can-i-use-a-search-results-table-to-power-anot.html

bkirk · ‎11-13-2018

Ok how about this I used makeresults in a subsearch to build the search and added the format to add the OR's: <fieldset submitButton="true"> <input type="multiselect" searchWhenChanged="true" token="Element_t"> <label>Element in Upper case</label> <choice value="*K21A* *DG23* *FM23*">K21A DG23 FM23</choice> <default>K21A DG23 FM23</default> <prefix>"</prefix> <suffix>"</suffix> <initialValue>*K21A* *DG23* *FM23*</initialValue> <valuePrefix></valuePrefix> <valueSuffix></valueSuffix> <delimiter> </delimiter> </input> And this is the search: <table> <search> <query>`Master_Search` [|makeresults |eval Element=$Element_t$ |eval Element=split(Element," ")|table Element | format "(" "" "" "" "OR" ")"]</query> </search> <option name="drilldown">none</option> </table> Keep in mind that the *'s are going to make the search slow but if these strings are in the middle of Element with no spaces before and after or special characters you will need them. I would leave the *'s out if you can. Also if these values need to be found in any part of the row not just in the Element you can change the Element to search: <table> <search> <query>`Master_Search` [|makeresults |eval search=$Element_t$ |eval search=split(search," ")|table search | format "(" "" "" "" "OR" ")"]</query> </search> <option name="drilldown">none</option> </table> Hope this helps you.

bkirk · ‎11-12-2018

Searching is generally case insensitive, so do you need to do all that changing to upper and creating a new field? I was able to do something like this below with the multiselect and just add it to my search with the Prefix/Suffix in the multiselect options. Also see the examples dashboard, they give a good example on how to use the multiselect. <input type="multiselect" searchWhenChanged="true" token="Element_t"> <label>Element in Upper case</label> <choice value="K21A">K21A</choice> <choice value="DG23">DG23</choice> <choice value="FM23">FM23</choice> <default>K21A,DG23,FM23</default> <prefix>(</prefix> <suffix>)</suffix> <initialValue>K21A,DG23,FM23</initialValue> <valuePrefix>Element="*</valuePrefix> <valueSuffix>*"</valueSuffix> <delimiter> OR </delimiter> </input> . . . <row> <panel> <table> <search> <query>`Master_Search` $Element_t$ |</query> </search> <option name="drilldown">none</option> </table> </panel> </row>

bkirk · ‎11-07-2018

This is what I added to the CIM and it worked I missed some quotes on the mvappend: if(index=bro,mvappend("QUERY","RESPONSE"),if(isnull(message_type) OR message_type="","unknown",message_type))

bkirk · ‎11-07-2018

Did you add this to the CIM for DNS resolution? I tried changing the Eval Expression in the CIM, and it breaks the datamodel. Error in 'eval' command: The arguments to the 'mvappend' function are invalid.

bkirk · ‎11-07-2018

I found my own answer! https://answers.splunk.com/answers/683453/how-do-i-create-a-multivalue-field-with-an-eval-fu.html

bkirk · ‎11-06-2018

The Detect Long DNS TXT Record Response does not show anything: | tstats count min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution where DNS.message_type=response AND DNS.record_type=TXT by DNS.src DNS.dest DNS.answer DNS.record_type | `drop_dm_object_name("DNS")` | eval anslen=len(answer) | search anslen>100 | `ctime(firstTime)` | `ctime(lastTime)` | rename src as "Source IP", dest as "Destination IP", answer as "DNS Answer" anslen as "Answer Length" record_type as "DNS Record Type" firstTime as "First Time" lastTime as "Last Time" count as Count | table "Source IP" "Destination IP" "DNS Answer" "DNS Record Type" "Answer Length" Count "First Time" "Last Time" Seems like all the Network_Resolution datamodel message_type and answers are unknown: answer record_type message_type count unknown TXT unknown 44058 unknown TXT unknown 25818 unknown TXT unknown 22868 unknown TXT unknown 17916 unknown TXT unknown 16092 unknown TXT unknown 16087 unknown TXT unknown 8159 I cut out the src and dest, but as you can see I would get nothing since all the message_type's are unknown and the search is looking for responses. I think every bro DNS log is both a query and a response if the DNS server responds to the query. Do I need to do something special to get the message type of response and the data in the answers? I know that there is data in the answers field for the index=bro sourcetype=bro_dns: _time query qtype_name answers rcode_name tag vendor 28:53.2 10.231.36.73.imwyyj2pluwatbkhz2yqgkzte3fhckp.r.mail-abuse.com TXT TXT 143 Mail from 73.36.231.10 blocked using Trend Micro Email Reputation database. Please see <http://www.mail-abuse.com/cgi-bin/lookup?\\73.36.231.10> NOERROR dns,network,resolution Bro Anyone else using Bro to get DNS into Splunk? Thank you, Brian

bkirk · ‎08-09-2018

So after fumbling around and getting the Splunk for Web Analytics app working, we were asked to make the sites base off the first part of the path of the URI. For example we have www.example.com/abc and www.example.com/xyz. These logs come from the same IIS server and the same log, but they want to treat these like 2 different sites. I tried modifying the generate sessions and generate pages to include the first part of the uri as the site and then added both sites to the website page: Site Host Source 1 www.example.com example.com C:\LogFiles\WebLogs\*.log 2 www.example.com/abc example.com C:\LogFiles\WebLogs\*.log 3 www.example.com/xyz example.com C:\LogFiles\WebLogs\*.log This is the logic I added to the Generate Sessions and pages to make the first part of the path part of site: | rex field=http_request "(?<sub_site>\/[^\/]+)(?<mod_request>\/.*)$" | fillnull sub_site value=""| eval site=site+sub_site | eval http_request=if(isnull(mod_request), http_request, mod_request) But then all 3 sites seem to have the same stats and nothing really appears when I try to select any of them in any of the dashboards. Beside having these sites put into separate log files is there any other way I can have multiple sites split out from the same host/log sources? Thank you, Brian

bkirk · ‎08-09-2018

Removing the CIM app and changing some data models we were able to get this to work. Doesn't seem practical if we need both CIM for other things and the splunk app for web analytics. Got it working but not really the answer I wanted.

bkirk · ‎08-06-2018

Has anyone else had Splunk CIM installed and not accelerated? It has a Web data model that seems to conflict with the Web data model in Splunk App for Web Analytics. To fix the issue we deleted the Splunk CIM app since we weren't using it on the search head the web analytics was installed on. Thank you, Brian Kirk

bkirk · ‎08-01-2018

So I have tried to load the sample data as well as some apache logs, yet when I go toe the Analytics Center I don't get any results for the site/time period. We have looked at the searches and it is looking for "Web.eventtype"=pageview however it seems that non of the eventtypes are there. Looking at the long search that creates the datamodel it has a DIRECTIVES function that seems to lose the eventtypes: DIRECTIVES(REQUIRED_TAGS(tags="pci,proxy,web_watchlist" intersect="t")). Are we doing something wrong, it seems like it should just work, but there is a lot that goes on behind the scenes to make this all happen and somewhere we seem to have a breakdown. I have done all the following: Created a custom index for my apache logs Index the sample set of data I have for one day of apache logs Configured the website Generated the sessions Generated the pages Enabled acceleration Note: we changed the sessions, pages, and datamodel to only search our index, weblogs_test to avoid pulling in other data we don't want to search yet. We added the index because we have other logs in splunk that also get the tag=web so we don't want to include those yet (30 gigs a day) in the datamodel until we get it working with this test data. Thank you, Brian Kirk

bkirk · ‎04-04-2018

Going off Pochichen, try this: (dest = 162.248.150.* OR 192.168.*.* ) AND src_ip != 192.168.*.* AND src_ip=184.168.152.52 | stats count by src_ip | sort -count |lookup whoisLookup ip AS src_ip| eval whois = substr(whois, 3, len(whois)-3)| spath path=WhoisRecord.RegistryData.Registrant.Country output=Country input=whois Or do get all the fields: (dest = 162.248.150.* OR 192.168.. ) AND src_ip != 192.168.. AND src_ip=184.168.152.52 | stats count by src_ip | sort -count |lookup whoisLookup ip AS src_ip| eval whois = substr(whois, 3, len(whois)-3)| spath input=whois

bkirk · ‎03-15-2018

I opted for this thanks for the |appendpipe that got me to my final answer! BASE_SEARCH | rex field=dest_host "^(?<hostname>([a-z0-9\.\-]*\.)?(?<Domain>[a-z0-9\-]{2,}(?=\.[a-z\.]{3,})\.(?<tld>[a-z\.]{3,})))" | stats sum(count) as Visited by hostname Domain tld | eventstats sum(Visited) as Vis1 dc(hostname) as Subdomains by Domain | eventstats sum(Visited) as Vis2 dc(hostname) as Sub2 by tld | dedup Vis1 Subdomains Domain tld | rename Vis1 as Visited | appendpipe [| dedup Vis2 Sub2 tld | rename tld as Domain | rename Vis2 as Visited | rename Sub2 as Subdomains] | table Subdomains Visited Domain | sort 0 - Subdomains Thank you!

bkirk · ‎03-14-2018

This was exactly what I wanted to do. Let me put a but out there 🙂 Some reason this is slower than my 2 searches, and the data is not the same for the same time period Top 3 results for yesterday the Domain results match but the TLD seem to not match but you definitely got me on the right track to simplifying my search and getting the output formatted: Mine: Subdomains Visited Domain 46678 19367036 com 8294 1574148 net 5554 177845 gstatic.com Yours: Subdomains Visited Domain 46724 19367128 com 8299 1574156 net 5554 177845 gstatic.com

bkirk · ‎03-14-2018

BASE_SEARCH | rex field=dest_host "^(?<hostname>([a-z0-9\.\-]*\.)?(?<Domain>[a-z0-9\-]{2,}(?=\.[a-z\.]{3,})\.(?<tld>[a-z\.]{3,})))" | stats count as Visited by hostname Domain tld | eventstats sum(Visited) as Visited dc(hostname) as Subdomains by Domain tld | dedup Visited Subdomains Domain tld | table Subdomains Visited Domain | append [search BASE_SEARCH | rex field=dest_host "^(?<hostname>([a-z0-9\.\-]*\.)?(?<Domain>[a-z0-9\-]{2,}(?=\.[a-z\.]{3,})\.(?<tld>[a-z\.]{3,})))" | stats count as Visited by hostname Domain tld | eventstats sum(Visited) as Visited dc(hostname) as Subdomains by Domain tld | dedup Visited Subdomains Domain tld | stats sum(Visited) as Visited sum(Subdomains) as Subdomains by tld | rename tld as Domain] | sort 0 - Subdomains You can see that I have basically the same search twice once on the getting me the totals for the domain and then again to append the totals for the TLD: Here is my results that I am trying to get: Subdomains Visited Domain 13 18 com 3 3 googleapis.com 2 4 google.com 2 2 doubleverify.com 2 2 net 1 4 microsoft.com 1 1 evidon.com 1 1 gstatic.com 1 1 ivaws.com 1 1 krxd.net 1 1 live.net 1 1 outlook.com 1 1 yahoo.com Here is another way to do it but I repeat all the TLD data for each domain so I don't like the output as much: BASE_SEARCH | rex field=dest_host "^(?<hostname>([a-z0-9\.\-]*\.)?(?<Domain>[a-z0-9\-]{2,}(?=\.[a-z\.]{3,})\.(?<tld>[a-z\.]{3,})))" | stats sum(count) as Visited by hostname domain tld | eventstats sum(Visited) as Vis1 dc(hostname) as Subdomains by domain | eventstats sum(Visited) as Vis2 dc(hostname) as Sub2 by tld | dedup Vis1 Subdomains domain tld | rename Vis1 as Visited | rename domain as Domain | table Subdomains Visited Domain Sub2 Vis2 tld Subdomains Visited Domain Sub2 Vis2 tld 3 3 googleapis.com 13 18 com 2 4 google.com 13 18 com 2 2 doubleverify.com 13 18 com 1 4 microsoft.com 13 18 com 1 1 evidon.com 13 18 com 1 1 gstatic.com 13 18 com 1 1 ivaws.com 13 18 com 1 1 krxd.net 2 2 net 1 1 live.net 2 2 net 1 1 outlook.com 13 18 com 1 1 yahoo.com 13 18 com So basically I would like to have the results of the appended searches using only the single search like did with the second search if possible. Thank you, Brian

bkirk · ‎09-08-2014

Thank you! It worked and thank you for the awesome explanation!

bkirk · ‎09-05-2014

I have 3 mail servers like so, 2 postfix servers and the last one not important Exchange, like so: Postfix1 -> Postfix2 -> Exchange Both Postfix server logs are in splunk. I see that when postfix1 hands email to postfix2 it correlates mail with a queued_queue_id How can I correlate the 2 sets of logs together in one transaction? Sep 5 11:04:27 nagaviper postfix/smtpd[21218]: B570F448E65: client=unknown[192.168.1.65] Sep 5 11:04:27 nagaviper postfix/cleanup[21216]: B570F448E65: message-id=<20140905150427.B570F448E65@nagaviper.123.com> Sep 5 11:04:27 nagaviper postfix/qmgr[20283]: B570F448E65: from=<autonotify@test.com>, size=1771, nrcpt=1 (queue active) Sep 5 11:04:27 nagaviper postfix/smtp[20729]: B570F448E65: to=<user@123.com>, relay=localhost[127.0.0.1]:10025, delay=0.18, delays=0.04/0/0.05/0.09, dsn=2.0.0, status=sent (250 OK: data received) Sep 5 11:04:27 nagaviper postfix/qmgr[20283]: B570F448E65: removed Sep 5 11:04:27 habanero postfix/smtpd[46622]: 8966F601B4: client=businessclass.comcast.net[11.22.33.44] Sep 5 11:04:27 habanero postfix/cleanup[52613]: 8966F601B4: warning: header Subject: Speed Alert!!! from businessclass.comcast.net[11.22.33.44]; from=<autonotify@test.com> to=<user@123.com> proto=ESMTP helo=<businessclass.comcast.net> Sep 5 11:04:27 habanero postfix/cleanup[52613]: 8966F601B4: message-id=<> Sep 5 11:04:27 habanero postfix/qmgr[56439]: 8966F601B4: from=<autonotify@test.com>, size=1418, nrcpt=1 (queue active) Sep 5 11:04:27 habanero postfix/smtp[52618]: 8966F601B4: to=<user@123.com>, relay=nagaviper.123.com[192.168.1.67]:25, delay=0.32, delays=0.27/0/0/0.04, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as B570F448E65) Sep 5 11:04:27 habanero postfix/qmgr[56439]: 8966F601B4: removed I initially had: | transaction queue_id keepevicted=true maxspan=60s| transaction message_id keepevicted=true maxspan=120s Until I found not all mail servers give a message_id and when it is blank as it was in this example it failed to correlate the entry. Thank you, Brian

bkirk · ‎07-11-2014

The Subject line doesn't give me the IP address of the mail server in a variable so I had to put a rex to extract it, and I had to add a search to the second search and the also I add conditions because I want to see servers that weren't blocked but are now. Thank you for your help in getting me to this answer! | multisearch [search sourcetype="postfix_syslog" NOT error_code=554 NOQUEUE | eval Type="Blocked"] [search sourcetype="postfix_syslog" NOT NOQUEUE Subject: NOT 127.0.0.1| rex field=_raw "Subject:(?<subject_new>.*) from \S+\[(?<ip_address>[^\]]*)"| eval Type="Allowed"] | rex field=from "@(?<fromdomain>.*)" | stats count(eval(Type="Blocked")) as countBlocked count(eval(Type="Allowed")) as countAllowed by fromdomain,ip_address,from | where countBlocked>0 AND countAllowed>0 | streamstats count as counter by fromdomain,ip_address | stats sum(countBlocked) as "Total Count-Blocked" sum(countAllowed) as "Total Count-Allowed" list(eval(if(counter<=5,from,null()))) as "Top Users" by fromdomain,ip_address | sort -countBlocked

bkirk · ‎07-11-2014

I slightly modified it but this got me to the right answer! Thank you, I will post my final search in another posting it doesn't leave me enough room in here.

bkirk · ‎07-11-2014

sourcetype="postfix_syslog" NOT NOQUEUE [IP_Address] Subject| stats count We log the subject for accepted emails and it contains the IP so I guess a count of these would give me number of successfully sent emails. The IP would have to be dependent the results of the first search say my first search returned: abc.com 1.1.1.1 100 me@abc.com,you@abc.com xyz.com 2.2.2.2 50 me@xyz.com,you@abc.com and say that 2.2.2.2 sent 10 successful emails that day but 1.1.1.1 sent 0 successful emails then I would only want to show xyz.com and filter out abc.com from my results.

bkirk · ‎07-11-2014

I have the following search to give me top email domains/servers that have been blocked by our email RBL. I would like to key in on the ones that were being accepted. So obviously I don't care about the spammer that has been trying to spam me all day but I want to know if someone who was sending me email and all of the sudden was getting blocked. sourcetype="postfix_syslog" NOT error_code=554 NOQUEUE | rex field=from "@(?<fromdomain>.*)" | stats count(from) as count by fromdomain,ip_address,from | sort -count | streamstats count as counter by fromdomain,ip_address | stats sum(count) as "Total Count" list(eval(if(counter<=5,from,null()))) as "Top Users" by fromdomain,ip_address | sort 25 -"Total Count" Basically what I would like to add to this search is a subsearch for the count of records for the same time period that have the same ip_address, but that weren't blocked. And then if the count of allowed records that weren't blocked are greater than the number that were blocked display only those records. Also if that is possible maybe also compare the prior time period for the allowed messages so if yesterday we were allowing gmail to send but today we aren't show those emails. Thank you, Brian

Posts	32
Solutions	5
Karma Given	11
Karma Received	12
Member Since	‎06-04-2014

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Microsoft DNS

Help please HDHomeRun

Can you help me a problem I'm having with BRO's DN...

Splunk for web analytics subsite

Analytics Center shows nothing even after the data...

combining stats question, can I do this as one sea...

How to correlate 2 sets of logs together in one tr...

Subsearch with results from my search

Splunk Python Script to Decode MIME Headers in ema...

Adding python script to search app

Microsoft DNS

Help please HDHomeRun

Re: How do I create an alert that triggers based o...

Re: Can you help us with understand how Splunk man...

Re: How do you search a field with multiple value...

Re: How do you search a field with multiple value...

Re: How do you search a field with multiple value...

Re: How do I create a multivalue field with an eva...

Re: How do I create a multivalue field with an eva...

Re: Can you help me a problem I'm having with BRO'...

Can you help me a problem I'm having with BRO's DN...

Splunk for web analytics subsite

Re: Analytics Center shows nothing even after the ...

Re: Analytics Center shows nothing even after the ...

Analytics Center shows nothing even after the data...

Re: How to parse xml results from the whois add on...

Re: combining stats question, can I do this as one...

Re: combining stats question, can I do this as one...

combining stats question, can I do this as one sea...

Re: How to correlate 2 sets of logs together in on...

How to correlate 2 sets of logs together in one tr...

Re: Subsearch with results from my search

Re: Subsearch with results from my search

Re: Subsearch with results from my search

Subsearch with results from my search