Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- Find Answers
- :
- About jacobpevans
jacobpevans
Motivator
Member since:
06-14-2019
08-28-2025
Community Statistics
| Posts | 293 |
| Solutions | 42 |
| Karma Given | 220 |
| Karma Received | 82 |
| Member Since | 2019-06-14 |
Activity Feed
- Karma Good Sourcetype Naming for mmccul. 05-23-2025 12:37 PM
- Got Karma for Re: What does this message mean regarding the health status of Splunkd?. 01-28-2025 05:32 AM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 09-23-2024 02:27 AM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 07-10-2024 01:05 AM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 02-25-2024 05:59 PM
- Got Karma for Re: How to format results and populate a dashboard dropdown filter with mcatalog?. 03-28-2023 03:28 PM
- Got Karma for Splunkd Warning: "C:\program files\...\local.meta already exists but with different casing: C:\Program Files\...\local.meta". 06-17-2022 01:10 PM
- Got Karma for Re: Why is my DB connect checkpoint value not updating?. 06-01-2022 02:18 AM
- Got Karma for Re: Different color for panel title. 03-21-2022 03:25 PM
- Got Karma for Re: How do you restart splunk?. 03-07-2022 05:46 AM
- Got Karma for Re: How to delete unwanted special characters in alphanumeric string?. 02-10-2022 05:01 AM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 01-17-2022 09:34 PM
- Got Karma for Re: What does this message mean regarding the health status of Splunkd?. 01-12-2022 01:11 PM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 11-03-2021 01:17 AM
- Karma Re: SimpleXML: Inherent brokenness regarding autorun and Submit button? for woodcock. 09-14-2021 04:26 PM
- Karma Re: How to use a scheduled report as base for dashboard? for niketn. 09-02-2021 05:24 AM
- Got Karma for Re: Different color for panel title. 08-15-2021 03:59 AM
- Got Karma for Re: The percentage of small of buckets is very high and exceeded the red thresholds...-When the red warning will disappear after fixed parsing?. 07-07-2021 11:18 PM
- Posted Re: Windows Event Logs monitoring on All Apps and Add-ons. 05-10-2021 11:13 AM
- Karma Re: How to avoid un-responsive dropdown input with 100K or more results using Simple XML for niketn. 03-18-2021 07:35 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 |
07-08-2020
05:28 AM
What later versions? What is the output of btool from my original response? What do you mean specifically by "it doesn't allow me"?
... View more
07-07-2020
09:47 AM
The first thing I would try is to simplify. You properly copied from the default `inputs.conf`, but since you did that, you only need to keep the lines that you are changing. So, try this first: ###### Splunk 5.0+ Performance Counters ######
## CPU
[perfmon://CPU]
disabled = 0
index=perfmon
## Logical Disk
[perfmon://LogicalDisk]
disabled = 0
index=perfmon
## Physical Disk
[perfmon://PhysicalDisk]
disabled = 0
index=perfmon
## Memory
[perfmon://Memory]
disabled = 0
index=perfmon
## Process
[perfmon://Process]
disabled = 0
index=perfmon
## ProcessInformation
[perfmon://ProcessorInformation]
disabled = 0
index=perfmon
## System
[perfmon://System]
disabled = 0
index=perfmon Restart your Splunk service. If you still don't see the other sourcetypes, we'll need to see your btool output. Navigate to `$SPLUNK_HOME\bin` and run this command: splunk cmd btool inputs list perfmon://CPU --debug Reply to this with that output if you haven't figured it out by this point.
... View more
06-11-2020
05:22 AM
Splunk Add-on for Microsoft Windows https://splunkbase.splunk.com/app/742/
... View more
- Tags:
- splunk_ta_windows
05-31-2020
10:16 AM
That's good to know. I wonder if running the health check might also fix it without the need for a restart.
... View more
05-28-2020
03:02 PM
I'm not sure. It seemed to be in the range of 24 hours, but I could never perfectly emulate it.
... View more
05-05-2020
09:24 AM
1 Karma
Did you ever hear anything back? I've revisited this numerous times, and we're about to go with @jkat54 's solution (custom one-time PowerShell input to update inputs.conf )
... View more
05-01-2020
09:49 AM
+1 for you.
Despite being old, this answer is still quickly found through Google and addressed my problem. The alternative option that I used based on this answer is to explicitly table out all expected fields instead.
So, with a limit of 100, rare_field will not show up with:
| index=yourindex
Even if you change "Interesting Fields" to "All Fields" (instead of >1% coverage). However, this search will return it where applicable:
| index=yourindex
| table rare_field
If you can identify all of your important fields, this will also work without changing limits.conf
Cheers,
Jacob
... View more
04-29-2020
01:51 PM
Since this is a top hit on Google, I will just add that Java was updated in the background unbeknownst to me. Our auto-updater puts the version in the directory name so I was able to take the above advice, update the Java path for DB Connect and restart to resolve the problem.
... View more
04-09-2020
05:10 PM
Great write-up @emcclure.
We are seeing the exact same splunk-powershell errors only on our Windows 10 and Windows Server 2016, although the forwarders do NOT crash like they do for you. Most of our affected servers do not have a - in the computer name, so I think this question actually has two different issues.
Versions:
Splunk: 7.2.4
Windows TA 6.0.0 (although there have been almost no changes from 6 to 8.0.0)
These are the two important lines in your log (mine too):
At C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell.ps1:280 char:1
$psDisposer.runspace.Close()
... View more
03-28-2020
04:56 PM
Add a custom value:
"All"; *
This will allow the user to select "All" and set the token to "*". Then, set this as the default.
... View more
03-28-2020
04:20 PM
Possibly related, our environment is virtually unable to run Qmulos - Compliance on Windows Server 2016. Clicking any "Submit"-style button takes 10-20 seconds to load regardless of the button's function. It is now an officially recognized bug. The problem is not on any other Windows OS versions.
... View more
03-20-2020
08:15 AM
@woodcock gave me an idea which did end up working. It has been running smoothly for months now. If you ever run into this issue, switch your searches to a reference to a saved search that runs every X minutes. While this is not optimal, this dashboard is used heavily in our environment so we are fine with it running the saved search in the background every 5 minutes. The simple XML looks like this:
<dashboard>
<search id="baseSearchName" ref="savedSearchName">
<refresh>60</refresh>
</search>
The panel part is identical to the one in my original question.
Thanks for the help everyone,
Jacob
... View more
03-11-2020
10:11 AM
It's hidden, but at the bottom of Luke's troubleshooting is the answer (http://lukemurphey.net/projects/splunk-website-monitoring/wiki/Troubleshooting).
Specifically,
Some of my inputs are returning a 403 error even though I haven't told them to use authentication.
This is likely because you have a proxy server configured but you input is unable to authenticate with the proxy. Oftentimes, this happens because there is a global HTTP proxy defined using using environment variables (often in $SPLUNK_HOME/etc/splunk-launch.conf).
To get around this, define a value for the proxy_ignore setting in website_monitoring.conf. You can also set this using the setup page for the Website Monitoring app. Set the value of the "Server Ignore List" to "*", like this:
All I did was set "Server Ignore List" to * via the GUI. Problem solved. My website_monitoring.conf looks like this:
[default]
proxy_type = http
thread_limit = 20
proxy_ignore = *
I would think this could be handled in the default website_monitoring.conf , but I'll leave that to the expert @LukeMurphey . Thanks for a great app! I just wanted to get this answer up since I do not see it anywhere else besides your troubleshooting wiki.
Cheers,
Jacob
... View more
03-11-2020
10:07 AM
@LukeMurphey
According to the issue at the URL below, this problem was resolved in 2.7.5, but I am on version 2.9.2.
https://github.com/LukeMurphey/splunk-website-monitoring/issues/31
Here's the error:
2020-03-11 12:46:05,880 ERROR Exception generated when attempting to get the proxy configuration stanza=web_ping://[web input name], see url=http://lukemurphey.net/projects/splunk-website-monitoring/wiki/Troubleshooting
Traceback (most recent call last):
File "D:\Program Files\Splunk\etc\apps\website_monitoring\bin\web_ping.py", line 941, in run_ping
self.get_proxy_config(input_config.session_key, conf_stanza)
File "D:\Program Files\Splunk\etc\apps\website_monitoring\bin\modular_input.zip\modular_input\shortcuts.py", line 31, in wrapper
return function(*args, **kwargs)
File "D:\Program Files\Splunk\etc\apps\website_monitoring\bin\web_ping.py", line 799, in get_proxy_config
website_monitoring_config = self.get_app_config(session_key, stanza)
File "D:\Program Files\Splunk\etc\apps\website_monitoring\bin\modular_input.zip\modular_input\shortcuts.py", line 31, in wrapper
return function(*args, **kwargs)
File "D:\Program Files\Splunk\etc\apps\website_monitoring\bin\web_ping.py", line 735, in get_app_config
raise Exception("Could not get the website_monitoring configuration")
Exception: Could not get the website_monitoring configuration
The steps I performed to achieve this:
- Install website monitor version 2.7.6
- Add a website input
- I believe it was broken here, but I did not verify this
- Upgrade to the latest website monitor (2.9.2)
... View more
03-09-2020
02:51 PM
1 Karma
Despite running as "Local System", apparently Splunk was unable to update the file. I relaxed the security on this checkpoint file only, and now it updates no problem.
2020-03-09 17:40:01.470 -0400 [QuartzScheduler_Worker-7] INFO c.s.d.s.dbinput.task.DbInputCheckpointRepository - action=save_checkpoint_success checkpoint=Checkpoint{value='9092006', appVersion='3.2.0', columnType=2, timestamp='2020-03-09T17:40:01.436-04:00'}
... View more
03-09-2020
02:49 PM
Howdy Splunkers,
After creating a new database input through the DB connect GUI, the query keeps returning the same values, and the checkpoint value is not updating.
Log search:
index=_internal host=[db_connect_host] sourcetype=dbx_server DbInputCheckpointRepository
Log messages:
2020-03-09 17:15:00.253 -0400 [QuartzScheduler_Worker-5] INFO c.s.d.s.dbinput.task.DbInputCheckpointRepository - action=load_checkpoint_from_cache checkpoint=Checkpoint{value='9090000', appVersion='3.1.2', columnType=2, timestamp='2020-03-09T17:06:32.535-04:00'}
2020-03-09 17:15:00.254 -0400 [QuartzScheduler_Worker-5] ERROR c.s.d.s.dbinput.task.DbInputCheckpointRepository - action=unable_to_save_checkpoint
java.io.FileNotFoundException: D:\Program Files\Splunk\var\lib\splunk\modinputs\server\splunk_app_db_connect\[db_connect_input_name] (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(Unknown Source)
at java.io.FileOutputStream.<init>(Unknown Source)
at java.io.FileWriter.<init>(Unknown Source)
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.dumpCheckpoint(DbInputCheckpointRepository.java:206)
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.saveImpl(DbInputCheckpointRepository.java:239)
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.saveCheckpoint(DbInputCheckpointRepository.java:115)
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.save(DbInputCheckpointRepository.java:102)
at com.splunk.dbx.server.dbinput.task.DbInputTask.saveCheckpoint(DbInputTask.java:114)
at com.splunk.dbx.server.dbinput.task.processors.HecEventWriter.writeRecords(HecEventWriter.java:68)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:79)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2020-03-09 17:15:00.254 -0400 [QuartzScheduler_Worker-5] ERROR org.easybatch.core.job.BatchJob - Unable to write records
com.splunk.dbx.server.exception.WriteCheckpointFailException: Error(s) occur when writing checkpoint.
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.saveCheckpoint(DbInputCheckpointRepository.java:121)
at com.splunk.dbx.server.dbinput.task.DbInputCheckpointRepository.save(DbInputCheckpointRepository.java:102)
at com.splunk.dbx.server.dbinput.task.DbInputTask.saveCheckpoint(DbInputTask.java:114)
at com.splunk.dbx.server.dbinput.task.processors.HecEventWriter.writeRecords(HecEventWriter.java:68)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:79)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Cheers,
Jacob
... View more
03-05-2020
12:54 PM
Happy to share something that took much longer than it should've. Thanks for marking as answer MuS, cheers.
... View more
02-15-2020
07:59 AM
You have incredible latency between the time events were actually indexed on your indexer(s) (_indextime) and the time that Splunk parsed out (the standard _time). Were you indexing historical logs, or are these logs supposed to be ingested in real-time?
The second answer does not make sense to me. If your _indextime and _time are that far off, I would not expect you to see the same gaps in both _time and _indextime. You may just have a few outliers that are throwing it off.
This will be difficult for us to help you with without more information. Could you share some sample events? This will show us your biggest outliers. Be sure to censor all sensitive information. The only information we really need are _indextime, _time, and the time in your _raw event from the search below:
index=<your index>
| eval latency_mins = round(abs(_indextime - _time)/60),
indextime = strftime(_indextime, "%Y-%m-%d %H:%M:%S")
| table _time indextime latency_mins index sourcetype _raw
| sort 0 - latency
If the time is at the beginning of the _raw event, you can add something like this to eliminate the sensitive information (change the 30 so that the full time is visible):
| eval _raw = substr(_raw, 0, 30)
Here's the documentation on what I'm trying to get at: https://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition
... View more
02-14-2020
12:43 PM
What is the output of this query? (Last 24 hours or whenever you have the missing data)
index=<your index>
| eval latency = abs(_indextime - _time)
| stats max(latency) as max_latency by sourcetype
| sort 0 - max_latency
Does this line chart have the same gaps you see in your original search?
index=<your index>
| eval _time = _indextime
| timechart count
... View more
01-31-2020
01:49 PM
I have the same problem using Get-ADObject and Get-ADComputer . I can't figure it out for the life of me.
In case this helps, official Splunk documentation says that PowerShell scripts must run as system (which is not what we're doing). However, I don't believe system has AD privileges.
https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsDatawithPowerShellscripts
Splunk Enterprise must run on Windows.
Splunk Enterprise must run as the Local System user to run all PowerShell scripts.
PowerShell v3.0 or later must be installed on the host.
Microsoft .NET version 4.5 or later must be installed on the host.
... View more
12-11-2019
12:35 PM
Don't forget to mark the answer if it helped you resolve your problem for others in the future.
... View more
12-02-2019
04:27 PM
Reference for the future - use the dashboard GUI search editor, then copy and paste from the XML generated for you. It's the easiest way if you have lots of HTML encoded characters.
... View more
12-02-2019
09:28 AM
Greetings @harshparikhxlrd,
This could be a number of things. First off, if you're using Splunk_TA_Windows (on your search head(s)), you should not need to rex out all of these values - it is done for you automatically.
Anyway, to your question, can you verify for me that they always return results? When you do where Auto = "SDA_E_AuditLog" and ... by log , Splunk will discard all null values (for Auto and log). To check for this, try the search below:
index=monitoring sourcetype="PEGA:WinEventLog:Application" (SourceName="RoboticLogging" OR SourceName="Application") (Type="Information")
| rex field=_raw "Automation=\"(?<Auto>.+?)\""
| where Auto = "SDA_E_AuditLog" OR isnull(Auto)
| rex field=_raw "Message=\"(?<log>.+?)\""
| eval log = if(isnull(log),"Unknown",'log')
| timechart dc(host) by log
Cheers,
Jacob
... View more
12-02-2019
09:22 AM
1 Karma
Greetings @harshparikhxlrd,
You are rounding in this line: | eval dur = round(((hh * 3600) + (mm * 60) + ss),2) , but then you take another average on this line: | stats avg(dur) as "Average Duration" by log, strr which will sometimes give repeating decimals.
You just need to round after the last average instead of before it, so your query should look more like this:
index=monitoring sourcetype=PEGA:WinEventLog:Application (SourceName="RoboticLogging" OR SourceName="Application") (Type= "Information") ("TaskID=\"Logs\"")
| rex "Automation=\"(?<Auto>.+?)\""
| where Auto = "SDA_E_AuditLog"
| rex "Message=\"(?<log>.+?)\""
| rex "Duration:(?<hh>\d+):(?<mm>\d+):(?<ss>\d+\.\d+)"
| eval dur = (hh * 3600) + (mm * 60) + ss
| rex "UserID=\"UNTOPR\\\(?<UID>.+?)\""
| eval User = host." : ".UID
| stats avg(dur) as "Average Duration" by log, User
| eval "Average Duration" = round('Average Duration',2)
| stats list(log) as Automation list("Average Duration") as "Average Duration (seconds)" by User
| table User Automation "Average Duration (seconds)"
Cheers,
Jacob
... View more
11-13-2019
02:34 PM
1 Karma
See @woodcock 's answer
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
08-28-2025
02:07 PM
|
Karma from
Karma given to
