Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- About hughkelley
hughkelley
Path Finder
Member since:
12-19-2010
07-24-2025
Community Statistics
| Posts | 56 |
| Solutions | 0 |
| Karma Given | 3 |
| Karma Received | 9 |
| Member Since | 12-19-2010 |
Activity Feed
- Posted Re: Splunk HEC receive data from Logstash http output on Getting Data In. 07-24-2025 03:15 PM
- Posted Using splunk-sdk in user-defined functions in Spark/Databricks notebook on Splunk Search. 10-30-2024 07:09 AM
- Tagged Using splunk-sdk in user-defined functions in Spark/Databricks notebook on Splunk Search. 10-30-2024 07:09 AM
- Tagged Using splunk-sdk in user-defined functions in Spark/Databricks notebook on Splunk Search. 10-30-2024 07:09 AM
- Tagged Using splunk-sdk in user-defined functions in Spark/Databricks notebook on Splunk Search. 10-30-2024 07:09 AM
- Got Karma for How to get docker system events (not container logs) into Splunk?. 03-13-2024 03:12 PM
- Posted Re: Filtering FirePower logs to reduce license usage on All Apps and Add-ons. 09-15-2023 04:50 AM
- Posted How to get docker system events (not container logs) into Splunk? on Getting Data In. 08-30-2023 05:27 AM
- Tagged How to get docker system events (not container logs) into Splunk? on Getting Data In. 08-30-2023 05:27 AM
- Posted Re: Attach CSV file to email using Sendresults on All Apps and Add-ons. 02-28-2023 04:32 AM
- Karma Re: Risk Event Timeline not working. Error: "Risk event has missing or invalid fields" for gabriel_vasseur. 02-15-2023 03:24 PM
- Posted Re: MaxMind DB Usage (more than just City) on Splunk Search. 11-28-2022 01:52 PM
- Got Karma for Re: Custom Alert Action - How to get search head DNS name?. 10-04-2022 01:29 AM
- Posted Re: Custom Alert Action - How to get search head DNS name? on Splunk Dev. 10-03-2022 05:53 PM
- Posted Adaptive Response Action - can we set a status of warning? on Splunk Dev. 10-02-2022 08:43 AM
- Tagged Adaptive Response Action - can we set a status of warning? on Splunk Dev. 10-02-2022 08:43 AM
- Tagged Adaptive Response Action - can we set a status of warning? on Splunk Dev. 10-02-2022 08:43 AM
- Posted Re: Capturing Risky Sign-ins from Azure into Splunk on All Apps and Add-ons. 09-29-2022 07:13 AM
- Posted Re: Custom Alert Action - How to get search head DNS name? on Splunk Dev. 09-28-2022 01:25 PM
- Posted Re: Add on Builder adds aob_events_in_meta.json and .aob_meta file- Has anyone encountered these extra files? on All Apps and Add-ons. 09-26-2022 10:00 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
07-24-2025
03:15 PM
There are a lot variables in the mix, but this worked for me when I needed to do a quick POC from a syslog-based source. input {
syslog {
port => 601
syslog_field => "syslog"
}
}
filter {
#Add time for Auto time extraction in Epoch for Splunk HEC
ruby { code => "event.set('time', event.get('@timestamp').to_i)" }
}
output {
# stdout { codec => rubydebug }
# file {
# codec => json_lines
# path => "E:\logstash-output\ZPA-records.json"
# }
http {
format => "json"
content_type => "application/json"
url => "https://http-inputs.XXX.splunkcloud.com/services/collector"
http_method => "post"
headers => ['Authorization', 'Splunk ${splunk_hec_token}']
mapping => {
time => "%{time}"
host => "ZPA-HEC"
source => "xxx"
event => "%{message}"
}
}
}
... View more
10-30-2024
07:09 AM
Background: I've created a small function in a spark/Databricks notebook that uses Splunk's splunk-sdk package. The original intention was to call Splunk to execute a search/query, but for the sake of simplicity while testing this issue, the function only prints properties of the service object. import splunklib.results as results
def get_useragent_string(hostname: str, srcports: int, destip: str, earliest:float):
return ("connected to %s" % service.info['host']) The function works fine when I call it from a Python cell in a notebook. get_useragent_string(hostname="XXX",srcports=49738,destip="104.16.184.241",earliest=1730235533) The function throws this exception when I try it from a SQL statement %sql
SELECT get_useragent_string('xxx', 49738, '104.16.184.241', 1730235533) Any idea what is different between Python and SQL execution? I think this is the salient part of the exception, but don't understand why this would be different in SQL vs Python. ^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1160, in _spliturl
if host.startswith('[') and host.endswith(']'): host = host[1:-1]
^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith' Full exception is below org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 574.0 failed 4 times, most recent failure: Lost task 0.3 in stage 574.0 (TID 1914) (10.244.27.5 executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/root/.ipykernel/141000/command-3792261566929826-3142948785", line 17, in get_useragent_string
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/client.py", line 3199, in oneshot
return self.post(search=query,
^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/client.py", line 941, in post
return self.service.post(path, owner=owner, app=app, sharing=sharing, **query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 322, in wrapper
except HTTPError as he:
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 76, in new_f
val = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 816, in post
response = self.http.post(path, all_headers, **query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1315, in post
return self.request(url, message)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1338, in request
raise
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1472, in request
scheme, host, port, path = _spliturl(url)
^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1160, in _spliturl
if host.startswith('[') and host.endswith(']'): host = host[1:-1]
^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:551)
at org.apache.spark.sql.execution.python.BasePythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:115)
at org.apache.spark.sql.execution.python.BasePythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:98)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:507)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage9.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$5(UnsafeRowBatchUtils.scala:88)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$3(UnsafeRowBatchUtils.scala:88)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$1(UnsafeRowBatchUtils.scala:68)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:62)
at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$2(Collector.scala:214)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:82)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:82)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:211)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:199)
at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:161)
at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:134)
at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:155)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.Task.run(Task.scala:102)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:1036)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:1039)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:926)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.$anonfun$failJobAndIndependentStages$1(DAGScheduler.scala:3998)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3996)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3910)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3897)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3897)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1758)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1741)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1741)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:4256)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4159)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:4145)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:55)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$runJob$1(DAGScheduler.scala:1404)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1392)
at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:3153)
at org.apache.spark.sql.execution.collect.Collector.$anonfun$runSparkJobs$1(Collector.scala:355)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
at org.apache.spark.sql.execution.collect.Collector.runSparkJobs(Collector.scala:299)
at org.apache.spark.sql.execution.collect.Collector.$anonfun$collect$1(Collector.scala:384)
at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
at org.apache.spark.sql.execution.collect.Collector.collect(Collector.scala:381)
at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:122)
at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:131)
at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:94)
at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:90)
at org.apache.spark.sql.execution.qrc.InternalRowFormat$.collect(cachedSparkResults.scala:78)
at org.apache.spark.sql.execution.qrc.ResultCacheManager.$anonfun$computeResult$1(ResultCacheManager.scala:552)
at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
at org.apache.spark.sql.execution.qrc.ResultCacheManager.collectResult$1(ResultCacheManager.scala:546)
at org.apache.spark.sql.execution.qrc.ResultCacheManager.$anonfun$computeResult$2(ResultCacheManager.scala:561)
at org.apache.spark.sql.execution.adaptive.ResultQueryStageExec.$anonfun$doMaterialize$1(QueryStageExec.scala:663)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1184)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$8(SQLExecution.scala:874)
at com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$7(SQLExecution.scala:874)
at com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$6(SQLExecution.scala:874)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$5(SQLExecution.scala:873)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$4(SQLExecution.scala:872)
at com.databricks.sql.transaction.tahoe.ConcurrencyHelpers$.withOptimisticTransaction(ConcurrencyHelpers.scala:57)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$3(SQLExecution.scala:871)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:195)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$2(SQLExecution.scala:870)
at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:855)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:157)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.IdentityClaim$.withClaim(IdentityClaim.scala:48)
at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.$anonfun$runWithCaptured$4(SparkThreadLocalForwardingThreadPoolExecutor.scala:113)
at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:51)
at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:112)
at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:89)
at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:154)
at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:157)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/root/.ipykernel/141000/command-3792261566929826-3142948785", line 17, in get_useragent_string
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/client.py", line 3199, in oneshot
return self.post(search=query,
^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/client.py", line 941, in post
return self.service.post(path, owner=owner, app=app, sharing=sharing, **query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 322, in wrapper
except HTTPError as he:
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 76, in new_f
val = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 816, in post
response = self.http.post(path, all_headers, **query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1315, in post
return self.request(url, message)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1338, in request
raise
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1472, in request
scheme, host, port, path = _spliturl(url)
^^^^^^^^^^^^^^
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-2b093fd4-a851-4495-b00b-d0a105a66366/lib/python3.11/site-packages/splunklib/binding.py", line 1160, in _spliturl
if host.startswith('[') and host.endswith(']'): host = host[1:-1]
^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:551)
at org.apache.spark.sql.execution.python.BasePythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:115)
at org.apache.spark.sql.execution.python.BasePythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:98)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:507)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage9.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$5(UnsafeRowBatchUtils.scala:88)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$3(UnsafeRowBatchUtils.scala:88)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$1(UnsafeRowBatchUtils.scala:68)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:62)
at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$2(Collector.scala:214)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:82)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:82)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:211)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:199)
at org.apache.spark.scheduler.Task.$anonfun$run$5(Task.scala:161)
at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:134)
at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:155)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.Task.run(Task.scala:102)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$10(Executor.scala:1036)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:1039)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:926)
... 3 more
... View more
09-15-2023
04:50 AM
Are you saying that you eliminate DNS logging from eStreamer only or that you don't log it at all (in the firewall)? I'm trying to find a way to keep logging it in Firepower but not pulling it into Splunk via eStreamer.
... View more
08-30-2023
05:27 AM
1 Karma
We have already enabled the Splunk logging driver, but this forwards logs from inside the containers. I want to capture the docker system-level events, as you would see from this command: docker events --filter event=stop --since '60m' https://docs.docker.com/engine/reference/commandline/system_events/ I see this app (not approved for cloud). Are there any other options? https://splunkbase.splunk.com/app/6113 https://github.com/quzen/docker_analyzer/blob/main/bin/docker_events.py
... View more
- Tags:
- docker
Labels
- Labels:
-
data
02-28-2023
04:32 AM
No, our users still paste the table into a spreadsheet when they need a file. I haven't looked back at the add-on in a few years (to see if they've added functionality).
... View more
11-28-2022
01:52 PM
In Splunk Cloud, CSVs are one way to go. We did this with the free ASN DB when we moved to cloud (couldn't get https://splunkbase.splunk.com/app/3531 for cloud). In short, it's a CSV-backed lookup with a CIDR match type over the column/field with the network range. We're also looking at https://splunkbase.splunk.com/app/3022 now.
... View more
10-03-2022
05:53 PM
1 Karma
This was suggested to me by a colleague (reading the email alert action link hostname) . This works nicely for me inside AOB code: from splunk.clilib import cli_common as cli
def process_event(helper, *args, **kwargs):
alert_actions_config = cli.getMergedConf("alert_actions")
alert_actions_hostname = alert_actions_config.get("email", {}).get("hostname", None)
if alert_actions_hostname:
helper.log_debug("Alert action hostname: {}".format(alert_actions_hostname))
... View more
10-02-2022
08:43 AM
I have an Adaptive Response Action (execute_flow in the pic below) that requires certain identity data about the subject of the notable (mobile phone number). Not all users have a mobile number set in ES identity. Currently, I throw a failure event in Python for this condition. Is it possible to return a warning status instead?
... View more
Labels
- Labels:
-
Python
09-29-2022
07:13 AM
I think this app will fill the gap @jconger describes: Microsoft Graph Security API Add-On for Splunk https://splunkbase.splunk.com/app/4564/
... View more
09-28-2022
01:25 PM
I'm afraid that returns the actual hostname, not the DNS name used for the UI and API. sh-0000000000.XXX.splunkcloud.com
... View more
09-26-2022
10:00 AM
I also get the aob_meta file when I export to SPL. This contains the test parameters I used during development, which feels like information leakage.
... View more
09-25-2022
03:42 AM
I'd like to purge my aob_py3 folder as much as possible so that I can put my add-on into source control. Browsing around GitHub, I see a wide range of aob_py3 contents. Some examples: https://github.com/eclecticiq/intelligence-center-app-splunk-ta/tree/main/TA-eclecticiq/bin/ta_eclecticiq/aob_py3 https://github.com/hRun/SA-haveibeenpwned/tree/master/bin/sa_haveibeenpwned/aob_py3
... View more
09-21-2022
05:37 PM
I was expecting to find a helper object function for this, but I don't see one here: https://docs.splunk.com/Documentation/AddonBuilder/4.1.1/UserGuide/PythonHelperFunctions I want to use the DNS name/URL of the search head in my alert action code. How can this be accessed (Splunk cloud, if it matters)? At least in my local test server, I see this, but it returns an IP address, not a DNS name. helper.settings["server_uri"]
... View more
06-07-2022
06:44 AM
I had to use a different calculation. The TZ is still off, but as I'm only looking for a date, it's close enough for my purposes. index=wineventlog source="WinEventLog:Security" EventCode=5136 LDAP_Display_Name=accountExpires Value>0
| eval expDate=floor((Value-116444736000000000)/10000000)
| dedup _time Account_Name LDAP_Display_Name expDate DN
| table _time Account_Name LDAP_Display_Name expDate DN
| convert timeformat="%Y-%m-%d" ctime(expDate) I still don't understand why my TZ is wrong. I assume the Value field is UTC and I'm subtracting the Splunk epoch (result of one-liner below) from that. ([datetime] "1970-01-01 00:00:00Z").ToFileTimeUtc()
... View more
10-27-2021
08:28 AM
We have been using WEF as our collection point for a while. We started out small but have expanded the range of events over time. We have ~5,000 hosts forwarding to a single collector. The collector is busy, but seems to be healthy based on conventional Windows indicators. However, we have some data loss between the centralized event and Splunk (cloud). Logs show up in the WEF collection log but never make it to the index. First, are there any performance tuning suggestions you can offer UF on a WEF collector? Second, can you think of any way to check on processing of a single event once it goes into the UF and heads to the indexer?
... View more
Labels
07-07-2021
09:02 AM
The add-on fails to line break JSON docs into separate events/logs when pulling from an event hub. Certain Azure services seem to write multiple JSON docs to a single event hub message. Is there an option to correct this parsing? {"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
{"body":{"records": {"DataCenterName": "East US 2", "DeploymentUnit": "xyz", "EventId": 160, "EventName": "AzureBackupCentralReport", "properties": {"VaultUniqueId": ".........
... View more
Labels
- Labels:
-
troubleshooting
06-09-2021
04:21 PM
Azure AD sign-in logs -> Azure event hub -> Splunk. Just make sure you're using v4.1.3 of the Splunk Add-on for Microsoft Cloud Services. Prior versions didn't handle event hubs properly. https://splunkbase.splunk.com/app/3110/
... View more
06-02-2021
02:34 PM
Neither of these ASN lookup apps is available in Splunk Cloud. Does anybody know of an alternative? ASN Lookup Generator https://splunkbase.splunk.com/app/3531/ SecKit for geolocation with MaxMind https://splunkbase.splunk.com/app/3022/
... View more
Labels
- Labels:
-
search
06-02-2021
02:27 PM
The latest version of the Splunk Add-on for Microsoft Cloud Services (4.1.3) reads from event hubs. You can send the non-interactive sign-in Azure logs to an event hub and then consume from there.
... View more
06-02-2021
03:20 AM
I see this message in the _internal log so I know the action was called, but I don't have any further clues about the issue. Invoking modular alert action=sendresults_alert for search="Stale AD Users" sid="scheduler_aGtlbGxleUBzb21wby1pbnRsLmNvbQ__search__......." in app="search" type="saved" The report also has a regular send email action, which did work. Running on Splunk Cloud
... View more
Labels
- Labels:
-
troubleshooting
05-27-2021
08:42 AM
I'm looking for the same. Based on this blog and my poking around the Graph API, I don't think they're easily accessible. https://www.michev.info/Blog/Post/3127/azure-ad-sign-in-logs-for-service-principals-and-other-recent-improvements I'm looking into the Log Analytics Space -> Splunk options now.
... View more
10-30-2020
04:20 AM
1 Karma
@hectorvp, we never fully got it working. This is where we ended up. The idea was to extract the SQL detail into the zzz field and then split it out from there. I'm still trying to remember what didn't work about this but can't find my notes. https://regex101.com/r/vrb7OK/1 #inputs.conf
renderXml = true
#props.conf
[SQL-Audit-via-xml]
# search time extractions
EXTRACT-xmleventdata = \<Data\>(?<zzz_sql_audit_data>.+?)\<\/Data\>
REPORT-000 = wel-col-kv-sqlaudit-atsearch
#transforms.conf
[wel-col-kv-sqlaudit-atsearch]
SOURCE_KEY = zzz_sql_audit_data
REGEX = ([a-z_]+):(.*?)[\r\n]+
FORMAT = $1::$2
... View more
01-16-2020
12:13 PM
How do you add fields to a collection?
I created my collection via API and filled it with data. Then later on I added some fields to the JSON docs. I also tried POSTing a new field list into the configuration.
the collections.conf file still has the original field list,
the lookup (based on the KV) shows new data/fields I've added since then
the REST API view of the schema has no fields at all:
name #text
---- -----
disabled 0
eai:acl
eai:appName search
eai:userName nobody
profilingEnabled false
profilingThresholdMs 1000
replicate false
replication_dump_maximum_file_size 10240
replication_dump_strategy auto
type undefined
... View more
- Tags:
- kvstore
Labels
- Labels:
-
kvstore
01-10-2020
09:27 AM
That bring us full circle. This URL:
https://xxx/servicesNS/nobody/search/storage/collections/data/networks/99.99.6.0%2f23
gets me this result:
WebCmdletWebResponseException
Could not find object.
The remote server returned an error: (404) Not Found.
I know the _key is in the collection because I do a full fetch of the collection at the beginning of the script. This works with other types of data (usernames and other _key data) - just not these addresses.
... View more
01-09-2020
08:48 AM
Unfortunately, I use the CIDR address as the _key value. This was necessary so that I didn't have to track a separate ID. When I use the query approach you describe I get the key in plain CIDR format.
netCidr : 99.99.6.0/23
_user : nobody
_key : 99.99.6.0/23
I'm thinking this means I need to find a supported way to encode it.
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
07-24-2025
03:13 PM
|
Karma from
