Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- About davidpaper
davidpaper
Contributor
Member since:
06-22-2011
02-28-2023
Community Statistics
| Posts | 126 |
| Solutions | 22 |
| Karma Given | 227 |
| Karma Received | 224 |
| Member Since | 06-22-2011 |
Activity Feed
- Got Karma for Re: How many pipelines should I use on a forwarder?. 09-11-2025 09:55 AM
- Got Karma for Re: How many pipelines should I use on a forwarder?. 07-21-2025 09:24 PM
- Got Karma for Re: How many pipelines should I use on a forwarder?. 05-22-2025 03:43 PM
- Got Karma for Re: How many pipelines should I use on a forwarder?. 10-31-2024 04:41 PM
- Got Karma for Re: What do I validate after I upgrade Splunk Enterprise to confirm the upgrade was successful?. 10-29-2024 06:20 AM
- Got Karma for Re: How many pipelines should I use on a forwarder?. 10-16-2023 10:32 AM
- Got Karma for Re: How many pipelines should I use on a forwarder?. 06-15-2023 12:38 AM
- Got Karma for Re: Has anyone set up automatic ingestion of PagerDuty Incident data into Splunk?. 06-14-2023 04:19 PM
- Got Karma for Re: How do I benchmark system health before a Splunk Enterprise upgrade?. 03-02-2023 02:05 PM
- Got Karma for How do I benchmark system health before a Splunk Enterprise upgrade?. 02-07-2023 11:28 PM
- Karma Re: Strange behaviour with | table * for DalJeanis. 11-09-2022 11:16 AM
- Got Karma for Finding search strings when all you have is an expired SID. 07-26-2022 05:32 PM
- Got Karma for How to detect duplicate GUIDs on forwarders?. 04-28-2022 09:38 AM
- Got Karma for Re: SmartStore Behaviors. 03-15-2022 11:36 AM
- Got Karma for Re: Why did ingestion slow way down after I added thousands of new forwarders and/or sources and sourcetypes?. 01-13-2022 01:29 PM
- Got Karma for Re: Why did ingestion slow way down after I added thousands of new forwarders and/or sources and sourcetypes?. 10-19-2021 08:03 AM
- Got Karma for Re: Why did ingestion slow way down after I added thousands of new forwarders and/or sources and sourcetypes?. 09-28-2021 11:04 PM
- Got Karma for Re: How to detect duplicate GUIDs on forwarders?. 09-20-2021 08:09 AM
- Got Karma for Re: What is included in HEC introspection data?. 09-08-2021 05:56 AM
- Got Karma for Re: How do I benchmark system health before a Splunk Enterprise upgrade?. 07-30-2021 08:10 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 3 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 2 | |||
| 5 | |||
| 1 | |||
| 0 | |||
| 2 |
04-18-2014
08:02 AM
I did go through the first time config. This is a distributed, clustered, SHP-enabled environment.
[splunk]:stmocprvsh1:/splunk/etc/apps/SA-nix/metadata$ more default.meta
# Application-level permissions
[]
access = read : [ * ], write : [ admin ]
export = system
[lookups]
export = system
The entries in default.meta look correct. Nothing has been overwritten as far as I can tell.
Looking in the manager -> All configurations, I can't find any of the 3 macros that show up in the error messages. I see them (manually) in the SA-nix/default/macros.conf, but Splunk doesn't can't.
... View more
03-18-2014
06:52 PM
I'm seeing the following red bar UI errors (also in the log files) when launching the Splunk for *NIX app version 5.0.1 with splunk version 5.0.2, http://splunk/en-US/app/splunk_app_for_nix/home:
Error in 'SearchParser': Could not find macro 'home_cpu_idle' that takes 2 arguments. Expecting stanza name 'home_cpu_idle(2)'.
Error in 'SearchParser': Could not find macro 'home_disk_used' that takes 2 arguments. Expecting stanza name 'home_disk_used(2)'.
Error in 'SearchParser': Could not find macro 'unix_noop' that takes 0 arguments. Expecting stanza name 'unix_noop'.
In digging around, I'm confused. I found the three macros, but they are in the SA-nix app, not the splunk_app_for_nix app. Looking at each of the macros in Manager, they are all owned by the SA-nix app, and not shared with any other app. I've got permission to view them (I'm logged in as a user in the admin user role), but from everything I know about permissions, this is not going to work.
How is the Splunk for *NIX app supposed to see the macros in the SA-nix app when they aren't shared?
The weirdest part about this is that when I installed splunk_app_for_nix and the SA-nix app on my Search Head, this worked for a while, as I remember being very frustrated trying to get data out of the UI, but all of the back end bits worked.
Any suggestions welcomed.
... View more
03-03-2014
05:56 PM
When you identify a Splunk instance as a component of a cluster, you either need to identify it as 1) cluster master process 2) indexing peer 3) search head.
At least through the GUI, there is no way I know of to make an indexer also a search head.
... View more
02-26-2014
06:25 PM
1 Karma
I'm not sure it is a bug either. A month averages 30 days, but if you are trying to compare calendar months, then there should be no expectation that they are always going to be equal. Comparing Jan to Feb has to come with understanding that one month is usually 3 days longer than the other, except when it's 2 days longer.
Maybe m=month (30 day variety) and r=real length month, which the length of the month varies by the month itself (Jan = 31, Feb 28 or 29, with calendar math involved to determine which), Mar = 31, et al).
12m = 360 days, 12r = 365/366 days depending on the year?
Crazy?
... View more
02-26-2014
12:44 PM
1 Karma
It appears that the timewrap (v1.6) thinks each month is 30 days.
Not every month is 30 days.
Any chance of this getting updated so month lengths are correct by the month?
... View more
- Tags:
- Timewrap
12-10-2013
04:24 PM
10 Karma
In the 5.x code (up to 5.0.2), there is a painful bug with CM. That bug is related to CM not persisting knowledge of frozen buckets across restarts (Bug ID: SPL-65100). This results in the CM kicking off dozens (hundreds? thousands?) of bucket fixup processes on the indexers. Until enough buckets are fixed up that the index becomes searchable again, there is a 100% search outage. Bad.
To speed up the recovery requires digging into the configs and altering 2 settings, based on the "size" of the indexers available. In our case, the indexers are 24 core, and can handle up to 24 concurrent fixup jobs and splunk-optimize processes. Since we are on a very fast SAN, there is almost no iowait for storage.
Note that you will want to adjust these settings based on the size/capacity of your indexer CPU core count & available IOPS. These processes can be very IO intensive.
The settings required to make this go faster are
On the CM, in /opt/splunk_clustermaster/etc/system/local/server.conf
[clustering]
max_peer_build_load = 23
On each of the indexers, and thus managed by the CM bundle in /opt/splunk_clustermaster/etc/master-apps/_cluster/local/indexes.conf
[default]
maxRunningProcessGroups = 23
Make sure the indexers are restarted to pick up this change. When the CM restarts, it will pick up the change. After a CM restart, bucket fixup will run massively parallel across all indexers and bury them. That's okay, because when there is a 100% search outage, USE ALL THE RESOURCES! to get search back online.
Disclaimer: Splunk support may not approve of these changes. Splunk support also doesn't have to take the lashing the local Splunk admin does when Splunk is unavailable for searching. Be careful. 🙂
... View more
12-10-2013
04:24 PM
4 Karma
Do you have Splunk 5.0, 5.0.1 or 5.0.2 and have chosen to run clustered peers? I bet you've noticed that when you have a lot of buckets (1000s) and the Cluster Master process restarts, that there is a 100% search outage for a while. Then, it magically fixes itself. It's not magic. It's Splunk! Why is this happening?
... View more
12-09-2013
08:12 PM
1 Karma
Also - customer created indexes don't show up in the clustering dashboard until there is an event in them! This confounded me for hours one day until I decided to send some data to the index, and lo and behold, as soon as there was a hot bucket for the index on one of my indexers, it immediately shows up in the clustering dashboard.
... View more
12-09-2013
08:09 PM
See my answer to this problem here: http://answers.splunk.com/answers/104772/how-to-restart-just-a-cluster-master/114460
It doesn't eliminate the problem, but helps the Cluster Master recover faster.
... View more
12-09-2013
08:06 PM
1 Karma
I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster_master_script_is_necessary
To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =
On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.
... View more
12-02-2013
08:04 PM
eventtype=mystats | fields _time, field1, field2, field3 | join _time [ eventype=mydata | tranaction .... | closed_txn = 0 | where _time >= "Set A _time - 5 seconds" AND _time <= "Set A _time + 5 seconds" | fields _raw ]
May give you exactly what you are looking for.
... View more
12-02-2013
07:28 PM
Ran into similar problems with 5.0.2. Backing the Deployment Server down to 5.0.1 solved them completely. No experience with 5.0.3 - 5.0.6 to know if the problems were completely fixed within the 5.x train.
... View more
11-18-2013
06:45 PM
What's the last thing you see in the splunkd.log before the ERROR TcpChannel messages?
... View more
11-18-2013
06:43 PM
For the TAs, once they are installed (per rsennett_splunk above), make sure that the TA is enabled. A TA that's installed but disabled won't do you much good.
... View more
07-26-2013
11:10 AM
4 Karma
Greetings,
I have a saved & shared search URL that has the SID in it. The search has long expired, and I'd like to get the original search string out of it.
Looking at: index=_internal $SID sort of works, but is painful to manually parse through. There really has to be a better way to do this.
A dead-sid search perhaps?
... View more
- Tags:
- sid
07-25-2013
10:56 AM
1 Karma
The issues with port conflict only appears if you run the DS and CM as separate splunk processes. If you run them both in the same process, then there is no port conflicts.
... View more
07-25-2013
06:43 AM
3 Karma
I have a DS and a Cluster Master running on the same host (a VM). I created them as separate Splunk installs (/opt/splunk_clustermaster and /opt/splunk_deploymentserver). As long as they don't conflict for ports and aren't in contention for CPU/RAM, there is no reason not to co-locate them for small to medium sized deployments.
When DS gets really busy, this may become a problem. Like jtrucks says, either back down the frequency of checkins for DS clients, or setup additional DS instances on other hosts.
... View more
07-06-2013
06:39 PM
1 Karma
Hi,
Maybe check out Splunk's Donut Charts ( http://splunk-base.splunk.com/apps/56187/donut-charts ) for your data. Might give you what you are looking for.
... View more
06-13-2013
01:20 PM
1 Karma
For a long backgrounded job, it would be really useful to be able to get an alert sent out when it is done. Doesn't appear that my Splunk instances does this.
Emails for scheduled searches that generate alerts work just fine, so I know the email server path is fully functional.
If it makes a difference, all of our user auth is done via LDAP.
Any suggestions on ways to get this to work?
... View more
04-29-2013
01:39 PM
I took the cheating route. It worked nicely, though I did end up going through 3 full rounds of clustered indexer restarts. This procedure should probably end up in public Splunk docs somewhere, as more people start using clustered indexers, they'll end up having to move indexes once in a while. Thanks!
... View more
04-28-2013
07:34 AM
3 Karma
Ahhh, the joys of running clustered indexers.
I need to find a way to move the _internaldb out from /opt/splunk/var/lib/splunk to /splunk/hot (how/warm buckets) and /splunk/cold (cold buckets) in a clustered environment.
The config is easy: in the cluster master's $SPLUNK/etc/master-apps/_cluster/local/indexes.conf I'll add an entry in there for the _internaldb and point it at the right place -- stealing the original config from $SPLUNK/etc/master-apps/_cluster/default/indexes.conf.
What I can't figure out is the mechanics of the move.
I have to make the change at the cluster master layer, and have it push the change out. But, if I do that, then the old _internaldb data will get orphaned in the old location unless it gets copied to the new space.
Should I take all 4 indexers down at once, move the _internaldb data to the new space manually, edit the config on each indexer manually and then restart them? Then update the config on the cluster master and re-push the bundle?
Not sure how to proceed here.
... View more
04-18-2013
10:25 AM
9 Karma
Hi,
To expand on yannK's answer excellent, there appears to be 2 config items in JSChart that would affect this, and they both live in $SPLUNK/share/splunk/search_mrsparkle/modules/results/flash/JSChart.conf.
1) [param:maxResultCount]
required = False
default = 500
I changed this to:
default = 5000
to allow jschart to pull in more data points per series.
2) [param:resultTruncationLimit]
required = False
default = 0
change this to something other than 0, so that for EVERY JSChart that gets rendered, it gets the benefit of the higher limit, instead of having to put that xml setting change in for every panel.
I set default = 20000 and at least with Google Chrome as the browser, it has no problems.
Hope this helps.
... View more
03-22-2013
08:28 AM
1 Karma
Restart WHICH splunk process?
The Deployment server or each one of the clients trying to talk to the Deployment server?
... View more
03-21-2013
03:04 PM
5 Karma
There appears to be a problem with the TS-sos addon when running in a Clustered indexer environment.
I see this error on all of my indexers:
03-21-2013 22:01:23.739 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/slave-apps/TA-sos/bin/ps_sos.sh" /bin/sh: /opt/splunk/etc/slave-apps/TA-sos/bin/ps_sos.sh: Permission denied
After doing some hunting, the permissions are right on the $Clustermaster/etc/master-apps/TA-sos/bin:
[splunk]:stmocprvsh1:/opt/splunk_clustermaster/etc/master-apps/TA-sos/default$ ls -l ../bin
total 12
-r-xr-xr-x 1 splunk splunk 2515 Jul 11 2012 common.sh
-r-xr-xr-x 1 splunk splunk 1445 Sep 19 2012 lsof_sos.sh
-r-xr-xr-x 1 splunk splunk 2075 Oct 4 05:38 ps_sos.sh
But when the bundle is created and pushed out the the Clustered indexers, permissions get changed:
[splunk@stmocprvidx3 hot]$ ls -l /opt/splunk/etc/slave-apps/TA-sos/bin/
total 12
-rw------- 1 splunk splunk 2515 Mar 21 21:23 common.sh
-rw------- 1 splunk splunk 1445 Mar 21 21:23 lsof_sos.sh
-rw------- 1 splunk splunk 2075 Mar 21 21:23 ps_sos.sh
I can seem to find no way to tell the Clustermaster not to change the permissions on files under the master-apps/ directory.
Anyone else see this?
-dave
... View more
10-25-2011
01:23 PM
2 Karma
It would be really useful to be able to set the default permissions (Private/App/All and Everyone Read/Write as examples). This would make creating Dashboards with saved searches a lot less error prone. It is frustrating when someone creates a search that shows up on a dashboard and has not remembered the manual step to make it visible to all.
... View more
- « Previous
- Next »
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
02-28-2023
01:16 AM
|
