After the Splunk Master enters maintenance mode, one of the indexers goes offline and then back online, and disables maintenance mode. The fixup tasks get stuck for about a week. The number of fixup tasks pending goes from around 5xx to 102 (after deleting rb bucket. I assume its the issue of bucket syncing in indexer cluster because client's server is a bit laggy(network delay, low cpu)) There are 40 fixup tasks in progress and 102 fixup tasks pending in the indexer cluster master. The internal log shows that all those 40 tasks are displaying the following error: Getting size on disk: Unable to get size on disk for bucket id=xxxxxxxxxxxxx path="/splunkdata/windows/db/rb_xxxxxx" (This is usually harmless as we may be racing with a rename in BucketMover or the S2SFileReceiver thread, or merge-buckets command which should be obvious in log file; the previous WARN message about this path can safely be ignored.) caller=serialize_SizeOnDisk Delete dir exists, or failed to sync search files for bid=xxxxxxxxxxxxxxxxxxx; will build bucket locally. err= Failed to sync search files for bid=xxxxxxxxxxxxxxxxxxx from srcs=xxxxxxxxxxxxxxxxxxxxxxx CMSlave [6205 CallbackRunnerThread] - searchState transition bid=xxxxxxxxxxxxxxxxxxxxx from=PendingSearchable to=Unsearchable reason='fsck failed: exitCode=24 (procId=1717942)' Getting size on disk: Unable to get size on disk for bucket id=xxxxxxxxxxxxx path="/splunkdata/windows/db/rb_xxxxxx" (This is usually harmless as we may be racing with a rename in BucketMover or the S2SFileReceiver thread, or merge-buckets command which should be obvious in log file; the previous WARN message about this path can safely be ignored.) caller=serialize_SizeOnDisk The internal log shows that all those 102 tasks are displaying the following error: ERROR TcpInputProc [6291 ReplicationDataReceiverThread] - event=replicationData status=failed err="Could not open file for bid=windows~xxxxxx err="bucket is already registered with this peer" (Success)"  Does anyone know what "fsck failed exit code 24" and "bucket is already registered with this peer" mean? How can these issues be resolved to reduce the number of fixup tasks? Thanks.   ... View more

Hello! I'm new to Splunk Cloud. Could you please explain the difference between hot, warm, cold and thawed buckets in Splunk Enterprise and Splunk Cloud? I understand that in Splunk Enterprise, a bucket moves through several states (from hot to thawed). However, when I click on a new index in Splunk Cloud, I only saw "Searchable retention (days)" and "Dynamic Data Storage". Does this mean that the amount of data that can be searched in the hot and warm buckets before it goes to cold is basically equal to the searchable retention (days)? Does Dynamic Data Storage basically equate to the Cold, Frozen and Thawed buckets (as in Splunk Enterprise)?   Furthermore, in the Splunk Cloud Monitoring Console, I can see DDAS and DDAA in the 'License' section. What exactly are these, and what is their relationship with data retention? What happens if the DDAS/DDAA exceeds 100%? Does this affect searching performance, or does Splunk Cloud simply not allow you to search data? Thanks. ... View more

I believe I have managed to get myself confused and would like to request assistance about field extraction. I have a new heavy forwarder, which is going to connect Splunk Cloud. First, the heavy forwarder will act as a simple Splunk Enterprise instance, before connecting to Splunk Cloud. The HF installed apps, such as  Fortinet Fortigate Add-on for Splunk,  Splunk Add-on for Palo Alto Networks,  Splunk Add-on for Microsoft Windows, Splunk Add-on for Checkpoint Log Exporter. I just simply installed and created inputs in local folder and they are good to go in HF. In the Splunk Enterprise instance, all inputs work fine. All fields are parsed properly, such as checkpoint logs, PA logs, Windows xml logs, fortigate logs. However, after connecting to Splunk Cloud, the universal forwarder credentials package is downloaded from Splunk Cloud and the app is installed in the HF. The connection is fine and logs are receiving. The weird issue is ONLY checkpoint and fortigate logs' fields are all extracted successfully, when I search in Splunk Cloud. For some reason, the Windows logs show a surprisingly small number of fields being extracted, when I search in Splunk Cloud. When I search the windows logs (old data in test index) in HF, it shows a LOT of interesting fields (>300), which is great. The PA logs only extracted host, index, source, sourcetype, _time (including default ones like linecount, punct, splunk_server), when I search in Splunk Cloud. I am confused because checkpoint and fortigate logs are all extracted successfully, but others are not. I understand that the apps are recommended to install across the deployment (https://docs.splunk.com/Documentation/AddOns/released/Overview/Wheretoinstall), but I would like to know a reason why some apps work and some apps do not. They are only installed in HF and the fields should be all extracted in the forwarder layer? Is it possible that the field extraction is not finished, since there are just too much data coming or too much data in total (PA logs >10000 events last 30 mins, windows logs >2000 events last 30 mins)? Thanks. I appreciate your help. ... View more

Hi. I am new to Splunk and SentinelOne. Here is what I've done so far: I need to forward logs from SentinelOne to a single Splunk instance. Since it is a single instance, I installed the Splunk CIM Add-on and the SentinelOne App. (which is mentioned in the Installation of the app. https://splunkbase.splunk.com/app/5433 ) In the SentinelOne App of the Splunk instance, I changed the search index to sentinelone in Application Configuration. I already created the index for testing purpose. In the API configuration, I added the url which is xxx-xxx-xxx.sentinelone.net and the api token. It is generated by adding a new service user in SentinelOne and clicking generate API token. The scope is global. I am not sure if its the correct API token. Moreover, I am not sure which channel I need to pick in SentinelOne inputs in Application Configuration(SentineOne App), such as Agents/Activities/Applications etc. How do I know which channel do i need to forward or i just add all channels? Clicking the application health overview, there is no data ingest of items. Using this SPL index=_internal sourcetype="sentinelone*" sourcetype="sentinelone:modularinput" does not show any action=saving_checkpoint, which means no data. Any help/documentation for the setup would be helpful. I would like to know the reason for no data and how to fix it. Thank you. ... View more

Hello there. I would like to ask about Splunk best practices, specifically regarding cluster architecture. One suggested practice is to configure all Splunk servers running Splunk Web (aka: a search head) as members of the indexer cluster, (at least that is what I hear from the architecture lesson). For example, there is a Splunk deployer. I need to use this command or achieved through web: splunk edit cluster-config -mode searchhead -manager_uri https://x.x.x.x:8089 (indexer cluster manager IP) -secret idxcluster Another one suggested practice is adding the Splunk servers (mention above such as deployers) to distributed search > search peers as well in manager. I would like to know why these are good practice and what are the benefits of doing these. (The deployer is not really a search head?) Thank you. ... View more

Hello. Im new at Splunk. Recently, I am trying to create and sign my own TLS certificates, following this official guide. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Howtoself-signcertificates However, splunkd.log keep on showing this error: Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL": Can't read key file /opt/splunk/etc/auth/mycerts/myServerCertificate.pem SSL error code=151441516 message="error:0906D06C:PEM routines:PEM_read_bio:no start line"   First, By following the guide, I created: private key of root certificate authority certificate, which is myCertAuthPrivateKey.key CSR for the certificate, which is myCertAuthCertificate.csr root certificate authority certificate, which is myCertAuthCertificate.pem Moreover, I created a server certificate and sign them with the root certificate authority certificate. private key for the server certificate, which is myServerPrivateKey.key CSR for the server certificate, which is myServerCertificate.csr Server certificate, which is myServerCertificate.pem   Basically, following the guide, i have 6 files in mycerts folder, and one srl file. This Splunk Master is a master node connects to 3 indexers (clustering). I followed this guide to modify the configuration files, which is the inputs.conf and server.conf i believe. Ref: https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/ConfigureSplunkforwardingtousesignedcertificates 6+1 files for certificate. /opt/splunk/etc/system/local/server.conf [general] ... [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem sslPassword = mypassword ... /opt/splunk/etc/system/local/inputs.conf [splunktcp-ssl:9997] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem sslPassword = mypassword requireClientCert = true sslVersions = *,-ssl2 Everytime i do service splunk restart, i still get the SSL error. Anyone know why and whats happening?? Same error is also happening in any other indexes. (same steps as i mentioned above) ... View more

Hello. I am completely new at Splunk. Recently, I've recently taken on a role where I'll be working with Splunk quite a lot. I have a question about SC4S (Splunk Connect For Syslog). I successfully installed the SC4S (podman + systemd) using the guide from this: https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/podman-systemd-general/ The SC4S is installed in Centos 7 VM (in vsphere). The HEC is configured successfully in heavy forwarder and I can successfully see the SC4S is properly communicating with Splunk. After that, I used Kiwi Syslog Message Generator from my windows 10 machine to send a syslog tcp message to the Centos 7 VM. Successful Output (TCP): However, if i sent a syslog udp message, the message was not successfully sent. As shown in the screenshot, the messages sent was zero after i pressed send. Unsuccessful Output (UDP): No new messages were shown in Splunk Web. 514 TCP and UDP is enabled in the firewall in Centos 7. I would like to request assistance about this issue. Thank you.                   ... View more