Hi @livehybrid , Thanks for sharing. This means the issue is caused by setting defaultGroup = thirdparty under [syslog]. In other words, all data is being sent to the third-party syslog destination by default. That’s why the settings in transforms.conf are being ignored, since the events are already routed to the third-party destination. For my current situation, I believe the configurations in props.conf and transforms.conf can remain unchanged, and only outputs.conf needs to be updated. #outputs.conf [tcpout] defaultgroup = indexer indexAndForward = 0 [tcpout:indexer] server = indexer_ip:8089 [syslog] # defaultGroup = thirdparty [syslog:thirdparty] server = thirdparty_ip:514 ... View more

I just realized that you are trying to send e.g. SH logs to indexers via HF, if I understand correctly? Why there is a this kind on order to add unnecessary complexity into your environment? Normal way is send all Splunk infra instances (like SH, LM, MC, CM, DS, Deployer etc.) log directly into indexers. This is actually 1st time when I even heard this kind of additional step/requirement! With this setup you will be probably get more issues than what this try to solve (I couldn't even guess what the problem is what you are trying to solve). But as said the issue is that indexers only listen to TLS enabled ports. This means that also those other Splunk Servers are trying to send TLS enabled streams. When you try to sent TLS enabled stream to HF (which just listen plain TCP), it didn't work. My proposal is that forget (at least SH, MC, LM, CM) to use HF between those and Indexers. If you have e.g. DS, HFs etc. outside of your main servers then those could be configured to use HF as IHF.  Then just ensure that those are using plain tcp instead of tcp with ssl in outputs. conf. ... View more

When you have cluster, then correct method is add nodes to it and after data has spread into those new nodes, then remove old nodes. Here is details how to do it https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platform-Linux-but/m-p/538062 ... View more

@ws  Absolutely. You'll need to update the outputs.conf file on all forwarders that send data to this server. Additionally, if this server is functioning as a deployment server, make sure to update the deploymentclient.conf file on the relevant clients as well. You can also consider using a DNS alias approach(depends on your environment) if you anticipate changing hostnames again in the future, without interrupting the forwarders splunk confs. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

Hi @ws  The clean command does not work on clustered indexes - See https://docs.splunk.com/Documentation/Splunk/latest/Indexer/RemovedatafromSplunk#How_to_delete:~:text=Note%3A-,The,-clean%20command%20does As the others have said, you could reduce the retention to basically nothing so that the data ages out, before then removing the indexes.conf stanza for the index and deploying out to your indexers. However note that this will not remove the old directory structure on the indexers for this index, if you want to completely remove it you will need to delete the folder structure on each node as per the docs "Once you've applied the indexes.conf changes and the peer nodes have restarted, remove the index's directories from each peer node." 🌟 Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing ... View more

Update I suspect this might be a Splunk-related issue, possibly due to the version I'm currently using (9.3.1). I spun up a new server for quick testing and reused the same configuration parameters from my previous setup. Mainly, the props.conf, transforms.conf, and inputs.conf. Interestingly, everything seems to be working fine on the new server, even though the configuration is identical to the old one. The only difference I can observe is in the data ingestion flow. initially, I ingested a set of JSON array entries in one format, and later ingested another set with a different structure containing more fields. So far, it all appears to be working as expected. However, when I tried the same method on my previous server, it didn’t work as expected. This is puzzling since both servers are using the same configuration files and setup. The only noticeable difference was the data ingestion flow on the new server, I ingested one format of JSON array first, followed by another with more fields, and it worked fine. But replicating this exact process on the older server doesn’t yield the same results. ... View more

Hi @livehybrid, i tried the following method to write into the local file with keeping the file at /tmp but it still didn't work. As for my situation, i think the best scenario would be keep a record of something like "seen before record.txt" and do a comparison and only to write new records into the file and remove previous indexed entries. At least the current approach is workable, but we’ll need to monitor the file size of "seen before record.txt" as it continues to grow. For now, the file size isn’t a concern since it only stores a limited number of tracking records. ... View more

Splunk (monitor input to be precise) doesn't care about the checksum of the whole file. It is obvious that the hash of the whole file will change as soon as _anything_ changes within the file. Whether it is a complete rewrite of the whole file contents or just adding a single byte at the end - the hash will change. The monitor input stores some values regarding the state of the file. It stores the initCrc value which will obviously change if the file is overwritten (and length of which can be manipulated in settings). But it also stores the seekCrc which is a checksum of the last read 256 bytes (and a position of those 256 bytes within the file). I suppose in your case the file ends by closing the json array, but after subsequent "append", the actual array is appended so its closing bracket is removed, another json structure is added and after that the array is closed in a new place. Unfortunately, you can't do much about it. As I said before - you'd be best off by scripting some external solution to read that array and dump its contents in a sane manner to another file for reading. ... View more

Anyone happen to know the following message? When i trigger a customize application, i get the follow message   ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c1106) ... View more

Hi @ws , you have many ways to check repetitive logs, the easiest is to save logs in a file with different names (e.g. adding data and time) and use the crcSalt = <SOURCE> option in the inputs.conf related stanza. Ciao. Giuseppe ... View more