Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- About twinspop
twinspop
Influencer
Member since:
06-09-2010
04-07-2021
Community Statistics
| Posts | 609 |
| Solutions | 72 |
| Karma Given | 384 |
| Karma Received | 232 |
| Member Since | 06-09-2010 |
Activity Feed
- Got Karma for Re: Why aren't defined field transformations showing in the GUI?. 09-09-2025 04:40 AM
- Got Karma for Re: Why is my log file sometimes ignored?. 05-19-2025 02:49 PM
- Got Karma for Re: How can I show the time of the last event in a stats by count table?. 12-11-2024 08:35 PM
- Got Karma for Re: Search to Identify when a host stops sending logs to Splunk. 10-09-2024 12:04 AM
- Got Karma for Re: Logging multiple sources from Docker with HEC: Are multiple sourcetypes possible?. 04-09-2024 06:41 AM
- Got Karma for Logging multiple sources from Docker with HEC: Are multiple sourcetypes possible?. 04-09-2024 06:41 AM
- Got Karma for Re: Logging multiple sources from Docker with HEC: Are multiple sourcetypes possible?. 04-09-2024 06:41 AM
- Got Karma for Re: How do I remove a dash (-) from the Account_Name field?. 02-22-2024 08:53 AM
- Got Karma for Re: How to configure props.conf for a Unix timestamp in a JSON log file?. 01-20-2024 09:06 AM
- Got Karma for Re: Why is my search head cluster captain logging KV Store replication errors?. 06-08-2023 12:21 PM
- Got Karma for Is there a way to convert a scheduled report to an alert? (6.6.3). 05-04-2023 03:51 PM
- Got Karma for Re: WinEventLog inputs: Why does current_only=1 skip server reboot events?. 03-15-2023 12:41 PM
- Got Karma for Re: How to change '_time'? _time is by default picking first alphabetical date column (Assigned) and doing +05:30. 09-30-2022 04:56 AM
- Posted Kafka add-on: Unable to initialize modular input on All Apps and Add-ons. 04-07-2021 07:52 AM
- Got Karma for Re: Is there any way to monitor CPU on Mac OS?. 03-20-2021 07:11 AM
- Got Karma for Re: With 8.1.2 or later should we still use an HEC tier on HWFs?. 03-08-2021 07:40 AM
- Posted Re: With 8.1.2 or later should we still use an HEC tier on HWFs? on Getting Data In. 03-05-2021 05:20 PM
- Posted With 8.1.2 or later should we still use an HEC tier on HWFs? on Getting Data In. 03-05-2021 05:19 PM
- Got Karma for Is there a way to convert a scheduled report to an alert? (6.6.3). 02-16-2021 11:23 PM
- Karma Re: Can you create/modify a lookup file via REST API? for 4n0m4l1. 12-15-2020 07:26 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
12-11-2016
09:03 PM
1 Karma
The btool command with the debug option might help:
splunk btool authorize list --debug | grep -v system/default
At least you can get a catalog of the settings you're targeting.
... View more
12-10-2016
11:29 AM
There is a REST endpoint for changing arbitrary config file properties.
curl -ku admin:changeme https://localhost:8089/services/properties/deploymentclient/target-broker:deploymentServer/targetUri -d value=your_splunk_ds:8089
Don't forget to restart after:
curl -ku admin:changeme https://localhost:8089/services/server/control/restart -X POST
... View more
12-08-2016
03:50 PM
Looks like the scheduling issue is a carryover from 6.5.0 that was not fixed. Hoping there's a workaround somewhere: https://answers.splunk.com/answers/456812/why-are-alerts-not-working-after-upgrade-to-splunk-1.html
... View more
12-08-2016
03:25 PM
Done. Still waiting for help. 😞 effectively down in the meantime.
... View more
12-08-2016
11:23 AM
Upgraded my clusters from 6.4.4 to 6.5.1 last night. Things appeared okay, but this morning 2 problems surfaced:
scheduled searches are not running on the SHC. If you open the saved search settings and click save, [EDIT: They show a schedule time, but don't actually fire.]
2/10 of our clustered indexers have filled queues. A restart of splunk gets things moving again for a few minutes, then back to full queues, blocked indexing. No errors are being logged. No indication of why they're blocked. Or why they work for 5-10 minutes, then stop.
Anyone else?
EDIT: mistaken description. Fake-editing the scheduled search gives it a "scheduled time" in the future, but it doesn't fire.
EDIT 2: Scheduling problems looks to be related to a known bug that was due to be fixed in 6.5.1, but apparently wasn't. https://answers.splunk.com/answers/456812/why-are-alerts-not-working-after-upgrade-to-splunk-1.html
EDIT 3: The problem referenced in EDIT 2 above was not related, although the error message was similar. See answer below.
... View more
12-06-2016
03:58 PM
My solution:
Created a collection (KV store) with one defined field, timestamp
Created the lookup for that collection
Created a saved search that runs every 4 hours...
... and sends the results to outputlookup
I use the userid as the _key, but I needed a way to isolate to a month. Eg, November ended on a Thursday. I don't want the data to be polluted or overwritten with December data before analysts had a chance to check. So I'm prepending every userid with the month:
sourcetype=my_sourcetype |
stats latest(_time) as timestamp by userid |
eventstats last(timestamp) as lasttime |
eval _key=strftime(lasttime,"%b")."-".userid |
fields _key timestamp |
outputlookup users_kv_lookup append=true
Now with a simple inputlookup command I can get uniques for the month of November:
| inputlookup users_kv_lookup where _key="Nov-*" | stats dc(_key) as uniques
I included the timestamp for... reasons. Maybe it will be useful to see the last login time. I thought having multiple logs for each user was too much though. We have 5m users logging in multiple times per day, each session with 10s - 100s of hits. That could get ugly.
EDIT - Dawg, do u even optimize?!
It seems like overwriting existing uids with the same info is time-consuming / resource-intensive. Why not only put the new ones into the collection? Okay then. Duuuh!
sourcetype=my_sourcetype |
stats latest(_time) as TS by userid |
eventstats last(TS) as lasttime |
eval _key=strftime(lasttime,"%b")."-".userid |
lookup users_kv_lookup _key |
where isnull(timestamp) |
fields _key TS | rename TS as timestamp |
outputlookup users_kv_lookup append=true
Any with existing records are excluded with the where isnull() clause. The speedup was at least 2 orders of magnitude. Woot woot.
... View more
12-03-2016
07:55 AM
I can't use estdc() because the data is not there. I need a uniques for the whole month, but only have about 19 days of retention. I'll have to run some tests to see how collect and/or the kvstore handles millions of records.
... View more
12-02-2016
12:10 PM
We have a few busy indexes that can only retain about 20 days worth of logs. The corner-office-types want unique user counts for the last month. The first thing that popped into my head was doing a daily stats count by user and collecting that into a summary index. But, I'm seeing upwards of 2m unique users each day. So that collect will be pretty massive.
Is there a better way? Maybe dumping that stats command into a kvstore?
At what point will collect fall over from too many lines of input?
Other considerations?
... View more
12-02-2016
09:11 AM
This situation really only applies if you are running a Search Head Pool. Which you shouldn't be doing. 🙂 The issue came about because SHP stores your Apps in a different location. So when I was upgrading splunk, it was updating the included apps in the default location, not the one actually used. Using tar to unpack the apps directory on top of the appropriate location should work: tar -xzf splunkforwarder-6.5.0-59c8927def0f-FreeBSD9-amd64.tgz --strip-components=2 -C /opt/splunkforwarder/etc splunkforwarder/etc/apps
... View more
12-02-2016
07:24 AM
Also, see here for more Metrics goodness:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Aboutmetricslog
... View more
12-02-2016
07:03 AM
Metrics logs are produced by all Splunk instances. If you have forwarders sending logs to an indexer, for example, you will potentially have 2 sets of metrics logs for any one category/series. You can get around this problem in 1 of 2 ways:
1) Isolate your search to your indexer(s) using host=your_indexer . The drawback here is that metrics are samples. By default, splunk only tracks the top 10 in each category. You can adjust this with maxseries under the metrics stanza in limits.conf.
2) Isolate your search to only the forwarders. One way to do this would be to EXCLUDE the indexers from your search: host!=your_indexer . Splunk still only tracks the top 10, but each forwarder has their own top 10, and you're more likely to get a complete(r) picture.
... View more
11-20-2016
02:46 PM
I built out a cluster in a test/lab scenario. Changing the RF and/or SF on the CM forces the change on all buckets, not just new ones. As far as I can tell. 🙂 The process above worked as planned.
... View more
11-19-2016
04:17 PM
I've now upgraded to 6.5.0 and have the same problem. With nearly 7000 users' worth of possible churn, I get to have a gold badge on all my search heads pretty much forever. Even after finding the culprits, which in itself is not easy, resolving them is such a pain in the ass.
... View more
11-18-2016
06:28 AM
Thank you, sir. I don't use indexer disco, but I don't think step 2 is vital. Correct? And step 3... are you saying that changing RF/SF has no effect on already stored data? If it's accurate, that's a surprise, but good to know!
... View more
11-17-2016
01:33 PM
3 Karma
We will be getting another batch of indexers in shortly, and each will have substantially more drive space than the old, 20x1.6TB SSD vs 20x600GB spinny. We will have more new servers than old. And we have approval to swap the drives in the old servers for new ones. (The h/w is otherwise the same. 20 core xeon, 128 GB. HP DL380G9.) Christmas is coming early. Woot woot.
Sketchy plan follows:
Increase RF and SF from RF=2/SF=1 to RF=3/SF=2
Wait for this move to settle
Shutdown an 'old' server
Slot in a new server to replace the old
Wait for Cluster magix; RF and SF met.
Optionally use 6.5.0's new goodness to rebalance data. Maybe?
Repeat 2-5 until all the old servers are OOS and new ones are in place
Replace drives in old servers
Add all old servers to the cluster as new members
Rebalance data one last time
Bask in the glory of a new indexer cluster with 140% more h/w and 500% more drive space. On SSDs.
(repeat entire process for our other cluster)
... View more
11-06-2016
09:17 PM
Yes. Cluster is 100% up. Previous captain reporting normally.
... View more
11-06-2016
05:21 PM
My SHC captain is reporting this:
Search peer <current_captain> has the following message: The search head cluster captain (https://<previous_captain>:8089) is disconnected; skipping configuration replication
I've since run a rolling restart, but the message persists.
... View more
11-03-2016
11:58 AM
Thanks for the input! But there is a problem with this approach. From the docs:
After you deploy an app to the
members, you cannot subsequently
delete the app's baseline knowledge
objects through Splunk Web, the CLI,
or the REST API. You also cannot move,
share, or unshare those knowledge
objects.
That's really, really not what I want. Hence the care to keep local and default separate in my steps above. I'm just wondering if I should copy the entire app into place before deploying. Or deploy first, then copy the non-default bits in.
... View more
11-03-2016
11:23 AM
1 Karma
I need to move a few apps from SHC1 to SHC2. My plan is below. Critique please!
(SHC1 uses deployer Dply1, SHC2 Dply2)
Stop all SHC members on SHC1
Copy target-app entirely from SHC1 to all members of SHC2. Move originals outside of $splunk
Copy target-app/default, target-app/metadata/default.meta and target-app/bin from SHC1 to Dply2 shcluster/apps. Move originals outside of $splunk.
Copy all users/target-app in their dir from SHC1 to Dply2 shcluster/users. Move originals outside of $splunk
Remove target-app from Dply1 shcluster/apps
Restart all SHC1 members
Apply shcluster bundle both deployers, Dply1 and Dply2
sacrifice goat or other farm animal
... View more
11-02-2016
03:43 PM
Try this:
(index=first OR index=second) (sourcetype=first-sourcetype OR sourcetype=second-sourcetype) |
chart count over FIELD1 by index |
eval diff=first-second
... View more
11-02-2016
10:16 AM
If you want something dynamic, you could use timechart + delta + stream stats. Something along these lines:
sourcetype=iis earliest=-6m@m latest=@m|
timechart span=1m dc(cs_username) as DU |
delta DU as Delta |
streamstats window=5 current=f global=t avg(Delta) as AvgLast5Delta |
eval PercentageChange=100*round((AvgLast5Delta-Delta)/AvgLast5Delta,4) |
tail 1
Alert on (absolute?) PercentageChange being bigger than some number.
EDIT: Made PercentageChange an actual percentage
... View more
10-31-2016
03:01 PM
Maybe second attempt above is what you're after
... View more
10-31-2016
02:41 PM
index=first OR index=second | stats count by sourcetype | fields count sourcetype
Maybe I'm misunderstanding the question.
EDIT: Second Attempt:
index=first OR index=second |
stats count by sourcetype field1 field2 field3 |
eventstats sum(count) as total by sourcetype |
fields total sourcetype field1 field2 field3
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
04-07-2021
02:44 PM
|
