About BryanScovill

BryanScovill · ‎02-13-2020

So, after a lot of digging and involving TAC and getting the case escalated we finally found a solution. It turns out my timelines were off and the root cause was an upgrade of the DB. Splunk's 5.1 driver was failing the handshake with MariaDB 8.0.17 so we went and manually upgrade the Connector/J to 8 and poof! Happy connections again. The folks at TAC were telling me that the 8 driver was still in testing as part of the DB Connect package and that package wasn't ready yet. They thought the driver update had a good chance of helping so we tried it manually. It was a real pain of a problem because the DB logs showed nothing at all unless we set debug to 4 and then only a handshake error not tied to any source. And handshake errors show up on other connections that are successful at that level of debug. Thanks.

BryanScovill · ‎01-15-2020

For some reason, after a VM event that caused a disruption to my Splunk's connection to downstream DBs, my MySQL connections will no longer connect. They report back "Could not create connection to database server" or ""Host 'XXX' is blocked because of many connection errors;" Connections to Postgres DBs stayed just fine. And, oddly, we can create a valid MySQL connection via the same server's OS level MySQL client. Logs on the DB side don't even suggest a connection attempt when we try via DB Connect, although the error sort of contradicts that. Logs on the Splunk side don't yield anything useful. All of the searching I've done points at a network level issues, but that doesn't appear to be the case. DB Connect and the MYSQL driver are all up to date and haven't been changed recently. Anyone have any thoughts on how to proceed? Thanks.

BryanScovill · ‎05-14-2019

Thanks all. The many hints got us there. Here is where we ended up. (index=firewall src_ip="" src_zone!=outside earliest=-24h ) OR (index=dhcp DHCP_ip="" OR DHCP_renewed="" earliest=-26h) | fields src_ip, src_zone, index, DHCP_ip, DHCP_renewed | eval dhcpip=coalesce(DHCP_ip, DHCP_renewed) | lookup dnslookup clientip AS src_ip OUTPUT clienthost AS src_host | search NOT src_host="*" | eval src_ip=coalesce(dhcpip,src_ip) | chart count by src_ip, index | where dhcp=0 Gracias!

BryanScovill · ‎05-13-2019

Wouldn't the second option just add the list from the subsearch to the the original search? I thought that was what append did. The first one seems promising, but "tables" is fighting with me. I am thinking that comparing lists via tables may be a no go, so I am experimenting with "fields"

BryanScovill · ‎05-13-2019

No Joy. I don't get why but data in the dhcp index still shows up. 😞

BryanScovill · ‎05-13-2019

I've been trying to research this for a couple of days and haven't been able to find anything just right. I am attempting to run a search (firewall source traffic with no DNS) and taking those results and searching our DHCP records so as to remove any DHCP addresses. Dynamic DNS ends up leaving those IPs nameless and my actual goal is to identify IP in use that haven't not been assigned (ie pilfered IPs) So, basically search1 & search2 generate lists of IPs and I want a list of IPs that are in search1 but not search2. This is one of many, many attempts... index=firewall src_ip="" src_zone!=outside | dedup src_ip | lookup dnslookup clientip AS src_ip OUTPUT clienthost AS src_host | search NOT src_host="*" | table src_zone,src_ip,src_host | join src_ip type=left [ search index=dhcp | eval src_ip=coalesce(DHCP_ip, DHCP_renewed), filter_out="y" | table src_ip, filter_out] | where not like(filter_out, "y") Any guidance would be appreciated.

BryanScovill · ‎12-15-2018

Well, I was trying to avoid join because it was so intensely slow over a log the size of a firewall log. I was trying to go down the route of a macro that I could feed ip & time and get a mac returned as a new field. I have been playing with the search above and a couple of thoughts. First is I am unclear why we'd go the stats realm. Is that returning the last instance of dhcp records with the dest_ip fed? Which leads me to the next thing... I need to go off of the start_time from the firewall log. The concern being dhcp logs for the same dest_ip after the start_time may reflect the dhcp address having been reassigned. Re: sample data... from the firewall. source_ip and start_time are bolded Dec 15 15:28:38 firewall-host 1,2018/12/15 15:28:38,,TRAFFIC,end,0,,10.21.94.190,192.229.210.163,132.177.238.65,192.229.210.163,rule 230,rmw1031,,incomplete,,Inside,Inet,ethernet1/19.384,ethernet1/20.100,,2018/12/15 15:28:38,,,51132,443,29213,443,,tcp,allow,286,146,140,4,2018/12/15 15:28:27,,any,0,,,10.0.0.0-10.255.255.255,United States,0,2,2,tcp-rst-from-client,0,0,0,0,,firewall-host,from-policy,,,0,,0,,N/A From the DHCP log. the dest_ip, dest_mac, and hostname are bolded. I've been using _time from this log. Dec 15 15:12:42 132.177.128.99 dhcpd[17818]: DHCPACK on 10.21.94.190 to c4:98:80:ee:45:64 (ConstanesiPhone) via eth3 relay eth3 lease-duration 7200 (RENEW) uid 01:c4:98:80:ee:45:64 My desired result would simply be to maintain the firewall log and append the dest_mac and hostname fields as new fields. But I should mention one of my struggles has been making sure the DHCP log entry is the latest entry before the firewall log's "start_time" field. I know that is one of the trippy bits.

BryanScovill · ‎12-14-2018

firewall does, but the DHCP logs only have an ip field (dest_ip) and the standard _time field. The _time field certainly wouldn't match the firewall's start_time but what I am shooting for is the the last DHCP log entry matching that IP before the start_time.

BryanScovill · ‎12-14-2018

Sorry, this is probably easy and I'm missing something, but but I've been beating my head on it so... I am trying to that the results from one search and take fields from that search and utilize them in a second search combining the results. Specifically, I am trying to search my firewall logs, take a source_ip and start_time and utilize those in a "data enrichment" type search against our DHCP logs to find mac & hostname. I've had some success with sub-searches, but that doesn't seem like the right mech. It seems like either a macro or a lookup but I'm dead ending on both. My latest attempt was with map... index=firewall | head 1 | map search="search index=dhcp DHCPACK dest_ip=$src_ip$ timeformat="%m-%d-%Y %h:%m:%s" latest=$start_time$ | | head 1 | fields dest_mac | fields - _time " ... which seems to eliminate all of my firewall results, leaving me with dhcp results. Any suggestions/pointers? Thanks

BryanScovill · ‎10-10-2018

We're struggling a bit with trying to use Webhooks instead of custom scripts in our alerts. Just as a simple test, we've created an alert to generate a post to one of our systems and instead of the JSON, all we appear to receive is "1". Are there any suggestions regarding the best way to test the received data? Is there anywhere that the payload is logged on the search head? I can see in the splunkd.log the event, but not the content... 10-10-2018 09:51:08.880 -0400 INFO sendmodalert - action=webhook STDERR - Sending POST request to url=https://redacted.supercool.address/test with size=5043 bytes payload The STDERR in there does raise my eyebrows. Any guidance would be appreciated.

Posts	10
Solutions	1
Karma Given	4
Karma Received	0
Member Since	‎11-03-2017

Online Status	Offline
Date Last Visited	‎08-26-2025 12:10 PM

Connection Error with DBConnect to MySQL

Comparing lists from two searches

Using results from one searchin a second and combi...

How do I view Webhook content in Splunk?

Re: Connection Error with DBConnect to MySQL

Connection Error with DBConnect to MySQL

Re: Comparing lists from two searches

Re: Comparing lists from two searches

Re: Comparing lists from two searches

Comparing lists from two searches

Re: Using results from one searchin a second and c...

Re: Using results from one searchin a second and c...

Using results from one searchin a second and combi...

How do I view Webhook content in Splunk?

Join the Conversation