Hi Folks; I came across this post on github https://github.com/kubernetes/kubernetes/issues/24677 and it had some fantastic options for pulling data from K8s/Docker into Splunk. It seems that the 'easy' approach here is to leverage the integration of K8S/Redhat with Fluentd, and then push the data into splunk. I was hoping to pick the brain of some of our Splunk experts to see if there is also a way to do a direct to splunk integration. Ideally, our goal is to make sure that the data that comes into splunk is 'containerized' so that it can easily be organized. I see the docker Splunk logging driver is available, but seems to be the less trusted approach since it doesn't integrate well with K8s. ... View more

I am having a VERY strange problem with my summary indexing. I have the following search running every hour at 20 minutes past the however, doing a summary of -1h@h to @h index="app_silayer7" "*~HTTP://*" NOT "*~ERR~*" |dedup _raw|bucket _time span=5m |rex field=_raw "^[^~\n]*~(?P<Domain>\w+)"  |rex field=_raw "^(?:[^~\n]*~){2}(?P<Service>[^~]+)"  |rex field=_raw "^(?:[^~\n]*~){3}(?P<Operation>[^~]+)"  |rex field=_raw "^(?:[^~\n]*~){4}(?P<NameSpace>[^~]+)" |rex field=_raw "^(?:[^~\n]*~){5}(?P<Consumer>[^~]+)" |rex field=_raw "^(?:[^~\n]*~){6}(?P<BackendTime>[^~]+)" |rex field=_raw "^(?:[^~\n]*~){7}(?P<TotalTime>[^~]+)" |rex field=NameSpace "(?<Version>[^\/])(|\/)$" |eval Version="V".Version |stats count as Count, avg(TotalTime) as TotalTime, min(TotalTime) as MinTime, max(TotalTime) as MaxTime, stdev(TotalTime) as STDDEV, perc95(TotalTime) as 95_Percentile ,by _time, Consumer, Domain, Service, Operation, Version Every once in a while however, I get some duplicate events. Some hours there are no duplicates, but some hours there are. The interesting thing is, on the hours that I see duplicates, I review the job results in the job inspector, and the results look clean! Has anyone run into this issue before? The raw events don't have any duplication in them, but it almost seems like when Splunk is stashing the results in my summary index, it hiccups and adds a few extra duplicate rows. For example, here are my results from the 9:00 hour: Service DetailCount SummCount delta Processor 36 72 36 Profile 55 110 55 ProfileAnd 185 370 185 It is exactly doubling the rows that were entered. ... View more

For those who have done some SNMP trap integrations with other monitoring tools, have you solved the issue of sending 'clear traps' when the condition is no longer met? For example, I have created a custom action that will send an SNMP trap to another one of my monitoring tools using trapgen. The integration works great, the only issue is that I need a way to send a clear trap for that alert when the condition is no longer met so that I dont have stale alarms in my monitoring tools. Wondering if anyone has done that yet? ... View more

Hi Folks; Wondering if someone could help me out here. I just had a big issue with Splunk. 3 of my Indexers just crashed for a bit (replication factor of 3). One of the services crashed with a bucket replication error (i fixed this), server 2 the service crashed and was simply restarted, server 3 completely got hung and required a reboot. After taking a quick peek, all of the stats are looking 'normal' including cpu/ physical/ storage, however, there was something that jumped out at me which was the iostats: Any particular reason this would start to happen? I just checked my forwarders and I dont see anything out of the ordinary with a large ramp in data ingestion I am working with my Linux team to restore one of my servers and they are stating that there was a "kernel level CPU soft lockup" Any Advice would be helpful in triaging this! ... View more

Hi Folks; I was wondering how to add some of the details that a user has put in for defining an Alert into the payload that gets sent to my custom alert. For example: Here is a sample alert that I am using. I have a custom app on my search head, and within the local folder there is an alert_actions.conf defined like so: [spectrum_alert] disabled=0 payload_format=json is_custom=1 icon_path=alerticon.png label=Enterprise Alert description=Dispatch Alerts to Command Center For Escalation within my app, there is a bin directory with a python script called 'spectrum_alert.py'. It looks like when the alert is triggered, two things are passed in, one being the '--execute' command, and second is the json payload that is passed in. There are however a few things missing that I would like to have, like the 'description', and the 'event count' for example. How would one add that? I know that with the out of the box command you can add things like $counttype$ $relation$ $quantity$, but is that still possible here with a custom alert? If so, could someone guide me? Thanks! ... View more