I need advice on troubleshooting SplunkHecExporter.  I'm using an OpenTelemetry Collector to accept logs via OTLP, export them to an on-prem Splunk Heavy Forwarder, which them forwards them to Splunk Cloud.  Below is my configuration.  I'm sending some test logs from Postman but the logs don't arrive in Splunk Cloud.  I see the arrival of the logs in the OpenTelemetry Collector through the debug exporter.  I confirmed connectivity to the Splunk Heavy Forwarder by setting an invalid token which results in an authentication error.  Using a valid token doesn't result in any debug logs being recorded.  Any suggestions on troubleshooting? exporters:   debug:     verbosity: normal   splunk_hec:     token: "<valid token>"     endpoint: "https://splunkheavyforwarder.mydomain.local:8088/services/collector/event"     source: "oteltest"     sourcetype: "oteltest"     index: "<valid index>"     tls:       ca_file: "/etc/otel/config/certs/ca_bundle.cer"     telemetry:       enabled: true     health_check_enabled: true     heartbeat:       interval: 10s service:   pipelines:     logs:       receivers: [otlp]       processors: []       exporters: [splunk_hec, debug]     telemetry:       logs:         level: "debug" ... View more

I'm having difficulty understanding why Query 2 is returning a different count than the other two queries. The cluster_name field is an indexed field. Does anyone know why this is occurring and have any suggestions to resolve or troubleshoot this issue? Query 1 index=infra_k8s* sourcetype="kube:objects:pods" | stats count by cluster_name cluster_name count ... clusterxyz 436 ... Query 2 index=infra_k8s* sourcetype="kube:objects:pods" cluster_name=clusterxyz | stats count by cluster_name cluster_name count clusterxyz 68 Query 3 index=infra_k8s* sourcetype="kube:objects:pods" | eval cluster=cluster_name | search cluster="clusterxyz" | stats count by cluster cluster count clusterxyz 436 ... View more

Is it possible to filter metrics on the Heavy Forwarder so they don't get passed along? Either a whitelist approach or blacklist approach is fine. ... View more

I have a set of events as follows for a chain of SQL Server blocked processes. It's a tree based structure. I am trying to join the data set on itself to determine which resources are blocking the most. Either result is okay but I prefer the first one. I'm able to figure out a solution for result #2 using a join and searching the same data twice. However, the number of events is more than 10K so it truncates the results. I have seen the selfjoin command in docs but am not certain how to join between two different fields in the same data set. Does anyone know how to produce either of the results below? Sample Events Process ID, Blocked By Process ID, Resource Name, Wait Time 1, 0, Resource 1, 0 2, 1, Resource 2, 15 3, 1, Resource 3, 10 4, 2, Resource 4, 5 Result set 1 Blocker, Total Blocked Victim Time Resource 1, 30 <- recursively sum the wait time Resource 2, 5 Result set 2 Blocker, Total Blocked Victim Time Resource 1, 25 <-- only sum the wait time of the children (not grandchildren, etc) Resource 2, 5 ... View more

I noticed in search.log that there are "INFO LookupOperator - Loading lookup table=..." log events that don't apply to the sourcetypes specified in the search. Later there is another event that says "INFO LookupOperator - Disabling automatic lookup of table=..." Why is Splunk loading lookup tables that aren't used by any of the source types specified in the search? ... View more

Does anyone know how to dynamically set the trendInterval option of a Single Value visualization equal to a time interval specified in a Time input? I believe I need to set the value equal to a token like below but am unsure how to set $mytoken$ equal to a time interval based on the range chosen in a Time input. However, I can't seem to get it to work. <input type="time" token="tf"> <change> <eval token="days">round(($tf.earliest$ - $tf.latest$)/86400,0)</eval> </change> <label></label> <default> <earliest>-4h@m</earliest> <latest>now</latest> </default> </input> ... &ltsingle&gt ... &ltoption name="trendInterval"&gt$days$d&lt/option&gt ... &lt/single&gt ... View more

I'm ingesting logs that have both event timestamps as well as timestamps within the contents of the logs. My props.conf contains BREAK_ONLY_BEFORE=<[A-Z] but it's breaking on CONTENTDATE as well. It is not exceeding the 10K default max event character length. Does anyone have any suggestions? &ltV ts="2018-07-16 22:14:28" &gt ... ... CONTENTDATE=2017-11-30 10:48:11 ... ... ... View more

I'm trying to parse out the exception type and exception message from the DB Connect dbx_server logs. I'm having some difficulty with this and have tried regex101.com and am able to come up with a regex that parses it but cannot get it to work using the rex command in Splunk. In the sample event below, I am trying to parse out "java.io.IOException" and "HTTP Error 503: Service Unavailable". From regex101 for the exception message - Exception: (?P.+? at) Event - 2018-07-05 16:21:05.528 -0400 [QuartzScheduler_Worker-21] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch java.io.IOException: HTTP Error 503: Service Unavailable at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112) at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89) at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36) at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203) at org.easybatch.core.job.BatchJob.call(BatchJob.java:79) at org.easybatch.extensions.quartz.Job.execute(Job.java:59) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 2018-07-05 16:21:05.528 -0400 [QuartzScheduler_Worker-21] ERROR c.s.d.s.dbinput.recordwriter.CheckpointUpdater - action=skip_checkpoint_update_batch_writing_failed java.io.IOException: HTTP Error 503: Service Unavailable at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112) at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89) at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36) at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203) at org.easybatch.core.job.BatchJob.call(BatchJob.java:79) at org.easybatch.extensions.quartz.Job.execute(Job.java:59) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ... View more