About joeldavideng

joeldavideng · ‎08-23-2021

I was wondering if anyone has successfully deployed a clustered instance of Splunk enterprise on AWS ECS Fargate. I'm looking to get rid of server management altogether for my cluster without having to go to Splunk Cloud. I read through a Splunk blog post on deploying a single instance to Fargate, but it didn't really cover things like dealing with Smart Store or the high memory requirements for indexers and search heads. If anyone has experience with this, I'd much appreciate the lessons learned.

joeldavideng · ‎10-23-2020

I recently started moving some of my indexes to Smart Store using AWS S3. I've noticed a lot of HTTP 204 and 404 errors in the Cache Manager when it tries to copy data from the local cache to the remote store. Strangely enough, the data seems to all be in the remote store and I'm not having any issues searching, but the number of errors is troubling. Does anyone know if the Cache Manager just normally misses write attempts and retries them until it succeeds?

joeldavideng · ‎05-22-2020

I am using Splunk Connect for Kubernetes on EKS which seems to capture most logs right out of the box. What it doesn't capture are interactive commands run from within containers. For example, if I "exec" into a container and run commands, none of that seems to be logged. Has anyone configured that level of auditing before or am I missing something in the default set up that should be capturing that?

joeldavideng · ‎04-17-2020

While setting up the Workday Add-On I started running into a certificate error when the app attempts to connect to the workday REST API. SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741) Has anyone else successfully set this up and know if that is an issue with my server configuration, the app, or something on the Workday side?

joeldavideng · ‎02-29-2020

Nick, Thanks for the insight. I had a feeling the real issue would be something having to do with when AWS region A turns back on. To clarify the why behind this, I have a business requirement to reduce the cost of Splunk and Smart Store seems like an obvious choice for an on-prem solution given that most of my costs are disk related. I also work in a heavily regulated industry and we have a requirement to simulate a regional AWS failure for a significant period of time regardless of AWS' actual up-time. Given the cost considerations, I don't think I'll be going with a muti-region cluster. However, given the eventually consistent replication of S3, I am going to need to rethink this approach.

joeldavideng · ‎02-28-2020

I am preparing to migrate my Splunk data storage to AWS S3 using Smart Store. My S3 buckets will be replicated across regions in AWS for failover and I have a requirement to fully test that capability. In theory, all of the data in my cluster should replicate to my failover region and I should be able to simply point my new Splunk instances to it and go. Has anyone ever tested this out and have insights into what will go wrong?

joeldavideng · ‎07-02-2019

The search "Threat - Source and Destination Matches - Threat Gen" is working and producing results, only the results aren't ending up in the threat_activity index where I would expect them to be. Does anyone know the dependencies I should be checking to ensure those events eventually end up in the Threat Activity dashboard?

joeldavideng · ‎05-28-2019

I think this does answer my question. The long term searches wouldn't be interactive and would be performed in the middle of the night once a month so I don't think the searches themselves would hose the index cluster. My main concern was what would happen when Splunk rolls the data from hot/warm to cold since there would now be a much larger disk to search for open space. Based on your answer it doesn't sound like that would necessarily be the main factor to worry about. I think I will try increasing the disk space and then increase the retention a month at a time to see if it becomes noticeable during the day. Thanks.

joeldavideng · ‎05-28-2019

I am currently running a 3 indexer cluster that provides 3 months hot/warm and 3 months cold retention for 120GB a day in AWS. With that setup I found I could technically extend my cold storage to 9 months with a modest increase in disk. My concern with doing this is that I've always heard you should expand Splunk horizontally rather than vertically. Given that the users who would be searching over a year of data would be doing so once a month, do you think that would put too much pressure on the cluster?

joeldavideng · ‎01-25-2019

Oh brother, I opened a support case days ago and headed here when they came up dry.

joeldavideng · ‎01-25-2019

I recently upgraded a Windows heavy forwarder to 7.2.3 and I am now getting errors when it attempts to connect to an indexer saying "Too many one time connections: Skipping". Has anyone ever encountered something like this and know what exactly the TcpOutProc service needs to get the data flowing again?

joeldavideng · ‎08-28-2018

I recently ran across some exploit kit modules designed to stymie incident responders by attacking endpoint security agents (Splunk included) and wanted to know if I could hide the Splunk service on my endpoints. The modules do simple string searches for service names so renaming the Splunk Forwarder service to something innocuous would do the trick. Does anyone know how to do this on Windows and Linux hosts without breaking Splunk?

joeldavideng · ‎08-15-2018

Straight from the horse's mouth. Thanks Starcher.

joeldavideng · ‎08-08-2018

I currently have several behavioral anomaly searches that report users exhibiting authentication behavior that is X number of standard deviations above their mean or just in the 9Xth percentile based on a baseline period of behavior. After reading George Starcher's blog entries on the subject of Extreme Search I am considering implementing those searches using that app, but I am concerned that I am losing precision because I can't say for certain that "anomalous" means three standard deviations above the mean. I would like to hear about other users' experiences doing the same and whether you think the migration is worth the effort.

joeldavideng · ‎05-28-2018

As it turns out, the answer is yes, Stream does in fact report DNS query events backwards. Each DNS record comes with two times; "timestamp", the start of the DNS session and "endtime", the end of the session. The default props.conf sets _time to "endtime", which produces the appearance of the backwards DNS events. For example, if a workstation makes a DNS request through an internal DNS server who then passes the request to an external server you would expect to see Workstation => Internal DNS Internal DNS => External DNS Internal DNS => External Authoritative DNS However, since the workstation technically has the longest time to wait for its session to complete it will appear as 1. Internal DNS => External Authoritative DNS 2. Internal DNS => External DNS 3. Workstation => Internal DNS

joeldavideng · ‎05-24-2018

Has anyone else found that their Stream DNS records report sessions backwards? I did some pretty extensive testing today with my SysAdmins and my Stream logs appear to show the initial DNS request from my workstation to the Domain Controller as the last event and the Domain Controller to the authoritative name server as the first event. The entire transaction makes sense if you reverse it, but the time stamps definitely show one event occurred 100ms before the other. I was thinking this may be an odd side effect of the way Stream collects DNS requests, but I'd like someone more knowledgable about the product to verify that assumption.

joeldavideng · ‎02-20-2018

This app is fire. Thanks a lot!

joeldavideng · ‎02-20-2018

I am running a daily query against an external database that provides a list of assets that many of my searches utilize. The searches exist on multiple search heads and I would like to avoid having to install DB Connect on all of them and replicate the expensive search. Is there a way to have the query run once on a centralized node and then have the others pull this list on a daily basis without having to use the deployment server?

joeldavideng · ‎02-13-2018

The Splunk Stream forwarders on my two primary DNS servers are failing every couple of hours for a TCP reassembly queue overflow. I tried doubling the queue size and filtering out all internal DNS traffic, but the agents still shutdown daily. Has anyone dealt with this before and know what settings to tweak?

joeldavideng · ‎02-13-2018

In the absence of any guidance I just doubled the count to 1 million. I can't say this solved my problem, but it the agents didn't complain.

joeldavideng · ‎02-08-2018

I have a couple of very high volume DNS servers that I am attempting to capture queries from using Splunk's Stream app. The Stream app fails within minutes of starting and throws an error stating the TCP reassembly queue has overflowed. Based on the fact that the Stream app works just fine on my low volume DNS servers and the content of the error message I believe I need to up the maxTcpReassemblyPacketCount. I checked out the documentation, which seems to define the default value, but I can't find a maximum value or anything stating what effect this will have on RAM usage. Does anyone know what the maximum value is and potentially a ratio of packets to RAM usage?

joeldavideng · ‎02-08-2018

Rayaivy, Create an app with an inputs.conf that defines the following [admon] disabled = 1 interval = -1 [MonitorNoHandle] disabled = 1 interval = -1 [WinPrintMon] disabled = 1 interval = -1 [perfmon] disabled = 1 interval = -1 [powershell] disabled = 1 interval = -1 [powershell2] disabled = 1 interval = -1

joeldavideng · ‎01-29-2018

Has anyone successfully integrated a Microsoft IIS SMTP server into Splunk? The logs the service creates are steps in a transaction rather than individual entries for actual emails sent. I wanted to normalize the log source to feed into the Email data model, but the logs are such a mess I am considering just deploying a different email relay that has better logging. I'd be particularly interested to know how to create single events that represent a sent email from the 10 or so individual rows that represent an email transaction. I realize you can use the transaction command, but I need an actual row to feed into a data model that can be accelerated.

joeldavideng · ‎12-27-2017

Sure thing, it was very close to the original. | datamodel Process_Monitoring Process_Creation search | eval threshold = [ | search index=summary "search_name=pm_command_line_length_stats" earliest=-90d@d latest=-1d@d | stats avg(command_line_length) AS command_line_average stdev(command_line_length) AS command_line_stdev | eval threshold = round(command_line_average + ( command_line_stdev * 6 )) | return $threshold ] | where $Process_Creation.command_line_length$ > threshold

joeldavideng · ‎12-26-2017

For anyone else with the same problem, it appears that doing comparisons against the actual value of a data model field requires you to use enclosing $s to get the value rather than the variable reference.

Posts	38
Solutions	4
Karma Given	8
Karma Received	6
Member Since	‎08-22-2017

Online Status	Offline
Date Last Visited	‎10-04-2021 06:32 PM

Is anyone running Splunk in AWS ECS Fargate?

Smart Store Producing a High Volume of 204 and 404...

How to Capture Kubernetes Interactive Exec Logging

Has anyone run into certificate validation issues ...

What will go wrong replicating smart store S3 buck...

Threat Intelligence Searches Not Populating threat...

Splunk App for AWS: Will my index cluster die if I...

Too many one time connections

Can you hide the Splunk Forwarder service to comba...

What is the best method of calculating variance, E...

Is anyone running Splunk in AWS ECS Fargate?

Smart Store Producing a High Volume of 204 and 404...

How to Capture Kubernetes Interactive Exec Logging

Has anyone run into certificate validation issues ...

Re: What will go wrong replicating smart store S3 ...

What will go wrong replicating smart store S3 buck...

Threat Intelligence Searches Not Populating threat...

Re: Splunk App for AWS: Will my index cluster die ...

Splunk App for AWS: Will my index cluster die if I...

Re: Too many one time connections

Too many one time connections

Can you hide the Splunk Forwarder service to comba...

Re: What is the best method of calculating varianc...

What is the best method of calculating variance, E...

Re: Is Stream DNS Reporting Sessions Backwards?

Is Stream DNS Reporting Sessions Backwards?

Re: Is there a way to propagate dbquery result to ...

Is there a way to propagate dbquery result to all ...

How do you fix a Stream TCP reassembly queue overf...

Re: What is a safe level for Stream's maxTcpReasse...

What is a safe level for Stream's maxTcpReassembly...

Re: How do I reduce the number of Windows 4688 eve...

How to Integrate Microsoft IIS SMTP into the Email...

Re: Calculated Data Model Field Value Inaccessible

Re: Calculated Data Model Field Value Inaccessible