About whar_garbl

whar_garbl · ‎02-19-2026

I can confirm the problem still exists on RHEL 9. I upgraded one of my clusters after discovering this problem and still have the same behavior.

whar_garbl · ‎02-10-2026

Great find, I can confirm this solved this (very annoying) problem for me!

whar_garbl · ‎02-05-2026

I don't see anything in the Enterprise docs about it - just Edge Processor (which I don't run). The Enterprise docs for 10.2.0 clearly list RHEL 8 as supported. Ref: https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/10.2/plan-your-splunk-enterprise-installation/system-requirements-for-use-of-splunk-enterprise-on-premises I get the impression that maybe whoever worked on your ticket saw that note about Edge Processor, assumed it was for everything, and jumped on the opportunity to wash their hands of it.

whar_garbl · ‎02-04-2026

I wonder if you haven't hit on the problem here. RHEL 8 only ships glibc 2.28 for its entire lifecycle. If there truly is a component requiring glibc 2.34 causing this problem, that seems like a huge oversight considering it's supremely impractical to push RHEL to a later glibc within the same major release. There is nary a mention of a glibc version requirement in the 10.2 system requirements, and RHEL 8 is listed as supported, so...

whar_garbl · ‎02-03-2026

I have the same issue post 10.2.0 and I'm seeing the same "stanza not found" messages. I also see in the logs lots of instances of trying (and failing) to phone home for updates, despite my best efforts to disable that globally.

whar_garbl · ‎03-04-2025

I have a file I'm monitoring that changes several times a day. It is likely that sometimes the file contents will be the same as a previous iteration, but not guaranteed (the file name name does not change). The file is in text format and is a few dozen lines long. I want to process the file every time the modtime changes, even if the content is 100% the same, and I want to create a single event with the contents each time. props.conf: [my_sourcetype] DATETIME_CONFIG = current BREAK_ONLY_AFTER = nevereverbreak [source::/path/to/file-to-be-read] CHECK_METHOD = modtime sourcetype = my_sourcetype inputs.conf: [monitor:///path/to/file-to-be-read] disabled = 0 sourcetype = my_sourcetype crcSalt = some_random_value_to_try_to_make_it_always_read If I update file-to-be-read manually by adding new lines to the end, it gets read in immediately and I get an event just like I want. But when the automated process creates the file (with an updated modtime), Splunk seems not to be interested in it. Perms are correct and splunkd.log reflects that the modtime is different and it's re-reading the file... but it doesn't create a new event. I'm sure I'm missing something obvious, but I'd appreciate any advice. Cheers.

whar_garbl · ‎03-18-2024

As title. I'm updating to UF 9.2.0.1 via SCCM, but a subset of targets are failing to install the update with the dreaded 1603 return code. The behavior is the same whether or not I run the msi as SYSTEM (i.e., via USE_LOCAL_SYSTEM) or not. All the existing forwarders being updated are newer - 8.2+, but mostly 9.1.x. Oddly, if I manually run the same msiexec string with a DA account on the local system, the update usually succeeds. It's baking my noodle why it will work one way but not another. I have msiexec debug logging set up, but it's not giving me anything obvious to work with. I can also usually get it to install if I uninstall the UF and gut the registry of all vestiges of UF, but that's not something I want to do on this many systems. I've read a bunch of other threads with 1603 errors but none of them have been my issue, as far as I can tell. Any ideas as to what the deal is?

whar_garbl · ‎08-31-2022

Correct me if I'm wrong here - I believe the right way to do the hack I'm talking about is to break the new cluster (it has no data or function as of now), then copy the buckets from the Old Indexer to one of the New (and now standalone) Indexers, then re-instantiate the cluster, which will then replicate the buckets to all its new best friends. It's not trivial to reconfigure the cluster and its associated SHC, though. I'm trying to avoid doing the eventtypes 'porkaround' because as you say, it will break a ton of searches, and I know the people who rely on those searches are not good at SPL; it will likely fall to me to cobble it back together. That will be time-consuming, to say the least. Reindexing is also not an option - that sounds like a nightmare, from my cursory reading.

whar_garbl · ‎08-31-2022

That's unfortunate. I have super long retention requirements, so I'd effectively be permanently keeping all the old and new data in their separate sources. I guess I'm going to go with the bucket copying hack. I don't like it, but I don't see a better way.

whar_garbl · ‎08-30-2022

I have a standalone instance with existing data on it. I have created a new indexer cluster that does not include this standalone machine. All instances are running the same OS and Splunk version. Can I add the existing data to the cluster by adding the standalone instance to the cluster as a peer? What will the behavior be in such a case? I'm aware of the bucket copying method, but I'm hoping there's a more hands-off method to accomplish this goal.

whar_garbl · ‎07-21-2022

Huh. It must be an issue of WHEN I crunch the case, because this does the trick and gets the results I expect. The only zero hosts now are true zeros. Success! You're the best! Thanks so much for sticking with this and holding my hand through it. I'm going to accept the post with the main query as the answer for the next person. Cheers!

whar_garbl · ‎07-20-2022

Expected: host count HOST1 11111 HOST2 22222 HOST3 33333 Actual: host count HOST1 11111 HOST2 22222 HOST3 0 host1 0 host2 0 host3 33333 Does that make sense? I get HOSTx and hostx (the lookups are all uppercased), and on one or the other there are zero events counted - but they're the same host.

whar_garbl · ‎07-19-2022

Ah, I see what you mean. Righto. So, then I end up getting the same results as explained, with ~10% of hosts showing up with different-case duplicates - one with appropriate event counts, and one with zero. Any thoughts on what's going on there?

whar_garbl · ‎07-19-2022

Thanks. I've tried a very similar permutation today, but this brings back the problem I had at another point: I get about a third more results than I should, and they're duplicates (one uppercase, one lowercase). If I cast the host field to upper or lowercase before the stats max command, I get the correct number of hosts but around 10% then erroneously return 0 events. E.g. | tstats count where [ | inputlookup file.csv ],index=* by host | append [ | inputlookup file.csv | eval count=0 ] | eval host=upper(host) | dedup host | stats max(count) as count by host Expected result: host count HOST1 12345 HOST2 67890 HOST3 24680 Actual result: host count HOST1 0 HOST2 67890 HOST3 24680 But if I search for HOST1 or host1 manually, there are many thousands of events in the same time period. Since field values are supposed to be case-insensitive (IIRC), I'm stumped as to why case is relevant here. It's trivial to do the case-casting in the actual lookup file if appropriate, but I think there's one more piece missing somewhere.

whar_garbl · ‎07-19-2022

This was close to what I want to do: | tstats count where [ | inputlookup file.csv ],index=* by host But it doesn't include hosts from the lookup that have no events, which I want to include. Adding a fillnull afterward doesn't do it.

whar_garbl · ‎07-18-2022

Thanks. I did start out using index=* as well, but I forgot about it. I still get null values for way too many hosts, but I can search for them directly using the same syntax (i.e. index=* and short hostname as found in the inputlookup) and get many thousands of events. It's making me nuts because it seems like it should be super straightforward!

whar_garbl · ‎07-18-2022

I have a lookup table with only one field, named host. The table contains a list of hostnames. I'm trying to find a way to get a count of events by host using this lookup table as the input (i.e. the hosts I want a count for). I've tried a variety of approaches. For example: |inputlookup file.csv | stats count by host Every host returns a count of 1. |inputlookup file.csv | join type=left host [|tstats count by host] About a dozen hosts return counts; the rest return null values. Complicating this problem seems to be case. If I crunch all the hosts to upper or lowercase, I get different results, but neither returns a complete result set. That seems super odd given that field values aren't case sensitive. I've tried crunching case with eval as well as in the lookup table itself, to no avail. We're stumped. What is the best approach to use a lookup table of hostnames to get an event count by host?

whar_garbl · ‎07-14-2022

Thank you for the detailed explanation. I'll try this in test and see how it works with my dataset. I suppose it isn't the end of the world if only newly indexed events have the short name, but I know my security folks will hate that answer. It'll be irritating to go back and edit every dashboard panel deployment-wide to accommodate the two-name dichotomy for historical searches.

whar_garbl · ‎07-14-2022

I have historical data in Splunk where the same host may appear as either Hostname.Domain.Com or Hostname. I would like all searches that specify Hostname to also gather events for Hostname.Domain.Com without modifying any searches. I can't delete and reindex, so that's right out. I found this post , which seems to be more or less what I want to do, but it isn't working, and I'm not sure why. It's older, so maybe the settings need to be different. What is the easiest way to accomplish this goal? Cheers.

whar_garbl · ‎10-06-2021

I have a UF on an rsyslog server. The UF is forwarding logs to the indexer successfully, but one of my two input flows is going to the wrong index, and I can't figure it out. inputs.conf [monitor:///path/number/one/*] index = first_index sourcetype = first_index host_segment = 4 disabled = false [monitor:///path/number/two/*] index = second_index sourcetype = second_index host_segment = 4 disabled = false Data of sourcetype second_index makes it to the corresponding index, but data of sourcetype first_index ends up in the main index. The only props and transforms I have configured are from the VMware add-on and its accessories, but I've scoured its conf files and have not found anything that would send this non-VMware data to main instead of where it belongs when it's specified in $SPLUNK_HOME/etc/system/local/inputs.conf. Any ideas? Thx!

whar_garbl · ‎10-05-2021

I figured it out. Someone had set a compliance baseline in SCCM to check and overwrite the conf files if they didn't match a specified value. Somehow that had been lingering for a couple of years without being disabled. Oops.

whar_garbl · ‎10-05-2021

Nope, I don't have anything like that.

whar_garbl · ‎10-05-2021

I have a single-instance Splunk setup with a handful of Universal Forwarders sending in data. There was previously a different architecture on this network, but this is a new build from the ground up - everything is new builds and fresh installs (all version 8.2.2.1; server is RHEL 8; clients are Windows 10). My UFs are installed with command line options to set the forwarding server and deployer (the same place). However, periodically, the clients' outputs.conf and deploymentclient.conf are being overwritten, and I cannot for the life of me figure out why. The settings being pushed in are for the old architecture, none of which remains on the network. Also, notably, it seems to only be the Windows UFs that are getting their settings overwritten - my *nix boxes do not appear to be affected as of now. I have attached a ProcMon to monitor the file edits. The changes are coming from splunkd.exe via the REST API: C:\Program Files\SplunkUniversalForwarder\bin\splunkd rest --noauth POST /services/data/outputs/tcp/server/ name=wrong_server.domain.com:9997 C:\Program Files\SplunkUniversalForwarder\bin\splunkd rest --noauth POST /services/admin/deploymentclient/deployment-client/ targetUri=wrong_deployer.domain.com:8089 I haven't yet found a way to manually elicit this change, and the update interval seems to vary from just a few minutes to every couple of hours. I've scoured my Group Policy and have not found any relevant settings there. I'm stumped. Does anyone have any ideas as to what may be doing this?

whar_garbl · ‎02-16-2021

I am rebuilding a SH cluster from scratch. I've followed the documentation carefully to this point. I have the shcluster captain bootstrapped and splunk show shcluster-status shows the captain as the only member, but the bootstrapping process failed to add my member nodes due to comms errors. Pretty sure I've got those fixed now. When I do splunk add shcluster-member -current_member_uri https://ip-address-of-captain:8089 on a member node, it tells me: current_member_uri is pointing back to this same node. It should point to a node that is already a member of a cluster. Obviously, I have checked and re-checked the uri, which I believe is correct (https://ip-address-of-captain:8089), and that is set right in server.conf on both sides. There is no IP conflict and the servers have no issue communicating. If I run splunk add shcluster-member -new_member_uri https://ip-address-of-member:8089 from the captain, it tells me: Failed to proxy call to member https://ip-address-of-member:8089 Google tells me this can be an issue with the pass4SymmKey, and to that end, I have updated the pass4SymmKey on both sides and restarted the instances a few times, to no avail. I'm stumped. Where did I go wrong that I can't get these search heads to cluster up nicely?

Posts	25
Solutions	1
Karma Given	14
Karma Received	3
Member Since	‎02-16-2021

Online Status	Offline
Date Last Visited	‎02-19-2026 12:56 PM

Reindex and create new event for file when modtime...

Updating Windows UF to 9.2.0.1 via SCCM - 1603 err...

Can I add the existing data to the cluster by addi...

How to find count of events by host from inputlook...

What is the easiest way to search-time transform f...

UF data going to wrong index

Output and deployment client configs constantly ov...

Search Head Cluster - can't add members after capt...

Re: After upgrade 10.0 to 10.2.0, Forwarder Manage...

Re: Spike in 503 errors in the Splunk WebUI after ...

Re: After upgrade 10.0 to 10.2.0, Forwarder Manage...

Re: After upgrade 10.0 to 10.2.0, Forwarder Manage...

Re: After upgrade 10.0 to 10.2.0, Forwarder Manage...

Reindex and create new event for file when modtime...

Updating Windows UF to 9.2.0.1 via SCCM - 1603 err...

Re: Can I add the existing data to the cluster by ...

Re: Can I add the existing data to the cluster by ...

Can I add the existing data to the cluster by addi...

Re: How to find count of events by host from input...

Re: How to find count of events by host from input...

Re: How to find count of events by host from input...

Re: How to find count of events by host from input...

Re: How to find count of events by host from input...

Re: How to find count of events by host from input...

How to find count of events by host from inputlook...

Re: Search-time transform for hostname and FQDN

What is the easiest way to search-time transform f...

UF data going to wrong index

Re: Output and deployment client configs constantl...

Re: Output and deployment client configs constantl...

Output and deployment client configs constantly ov...

Search Head Cluster - can't add members after capt...

Join the Conversation