I believe I have managed to get myself confused and would like to request assistance about field extraction. I have a new heavy forwarder, which is going to connect Splunk Cloud. First, the heavy forwarder will act as a simple Splunk Enterprise instance, before connecting to Splunk Cloud. The HF installed apps, such as  Fortinet Fortigate Add-on for Splunk,  Splunk Add-on for Palo Alto Networks,  Splunk Add-on for Microsoft Windows, Splunk Add-on for Checkpoint Log Exporter. I just simply installed and created inputs in local folder and they are good to go in HF. In the Splunk Enterprise instance, all inputs work fine. All fields are parsed properly, such as checkpoint logs, PA logs, Windows xml logs, fortigate logs. However, after connecting to Splunk Cloud, the universal forwarder credentials package is downloaded from Splunk Cloud and the app is installed in the HF. The connection is fine and logs are receiving. The weird issue is ONLY checkpoint and fortigate logs' fields are all extracted successfully, when I search in Splunk Cloud. For some reason, the Windows logs show a surprisingly small number of fields being extracted, when I search in Splunk Cloud. When I search the windows logs (old data in test index) in HF, it shows a LOT of interesting fields (>300), which is great. The PA logs only extracted host, index, source, sourcetype, _time (including default ones like linecount, punct, splunk_server), when I search in Splunk Cloud. I am confused because checkpoint and fortigate logs are all extracted successfully, but others are not. I understand that the apps are recommended to install across the deployment (https://docs.splunk.com/Documentation/AddOns/released/Overview/Wheretoinstall), but I would like to know a reason why some apps work and some apps do not. They are only installed in HF and the fields should be all extracted in the forwarder layer? Is it possible that the field extraction is not finished, since there are just too much data coming or too much data in total (PA logs >10000 events last 30 mins, windows logs >2000 events last 30 mins)? Thanks. I appreciate your help. ... View more

Hi. I am new to Splunk and SentinelOne. Here is what I've done so far: I need to forward logs from SentinelOne to a single Splunk instance. Since it is a single instance, I installed the Splunk CIM Add-on and the SentinelOne App. (which is mentioned in the Installation of the app. https://splunkbase.splunk.com/app/5433 ) In the SentinelOne App of the Splunk instance, I changed the search index to sentinelone in Application Configuration. I already created the index for testing purpose. In the API configuration, I added the url which is xxx-xxx-xxx.sentinelone.net and the api token. It is generated by adding a new service user in SentinelOne and clicking generate API token. The scope is global. I am not sure if its the correct API token. Moreover, I am not sure which channel I need to pick in SentinelOne inputs in Application Configuration(SentineOne App), such as Agents/Activities/Applications etc. How do I know which channel do i need to forward or i just add all channels? Clicking the application health overview, there is no data ingest of items. Using this SPL index=_internal sourcetype="sentinelone*" sourcetype="sentinelone:modularinput" does not show any action=saving_checkpoint, which means no data. Any help/documentation for the setup would be helpful. I would like to know the reason for no data and how to fix it. Thank you. ... View more

Hello there. I would like to ask about Splunk best practices, specifically regarding cluster architecture. One suggested practice is to configure all Splunk servers running Splunk Web (aka: a search head) as members of the indexer cluster, (at least that is what I hear from the architecture lesson). For example, there is a Splunk deployer. I need to use this command or achieved through web: splunk edit cluster-config -mode searchhead -manager_uri https://x.x.x.x:8089 (indexer cluster manager IP) -secret idxcluster Another one suggested practice is adding the Splunk servers (mention above such as deployers) to distributed search > search peers as well in manager. I would like to know why these are good practice and what are the benefits of doing these. (The deployer is not really a search head?) Thank you. ... View more

Hello. Im new at Splunk. Recently, I am trying to create and sign my own TLS certificates, following this official guide. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/Howtoself-signcertificates However, splunkd.log keep on showing this error: Error setting up SSL for TCP data input from file=inputs.conf stanza="SSL": Can't read key file /opt/splunk/etc/auth/mycerts/myServerCertificate.pem SSL error code=151441516 message="error:0906D06C:PEM routines:PEM_read_bio:no start line"   First, By following the guide, I created: private key of root certificate authority certificate, which is myCertAuthPrivateKey.key CSR for the certificate, which is myCertAuthCertificate.csr root certificate authority certificate, which is myCertAuthCertificate.pem Moreover, I created a server certificate and sign them with the root certificate authority certificate. private key for the server certificate, which is myServerPrivateKey.key CSR for the server certificate, which is myServerCertificate.csr Server certificate, which is myServerCertificate.pem   Basically, following the guide, i have 6 files in mycerts folder, and one srl file. This Splunk Master is a master node connects to 3 indexers (clustering). I followed this guide to modify the configuration files, which is the inputs.conf and server.conf i believe. Ref: https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/ConfigureSplunkforwardingtousesignedcertificates 6+1 files for certificate. /opt/splunk/etc/system/local/server.conf [general] ... [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem sslPassword = mypassword ... /opt/splunk/etc/system/local/inputs.conf [splunktcp-ssl:9997] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem sslPassword = mypassword requireClientCert = true sslVersions = *,-ssl2 Everytime i do service splunk restart, i still get the SSL error. Anyone know why and whats happening?? Same error is also happening in any other indexes. (same steps as i mentioned above) ... View more

Hello. I am completely new at Splunk. Recently, I've recently taken on a role where I'll be working with Splunk quite a lot. I have a question about SC4S (Splunk Connect For Syslog). I successfully installed the SC4S (podman + systemd) using the guide from this: https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/podman-systemd-general/ The SC4S is installed in Centos 7 VM (in vsphere). The HEC is configured successfully in heavy forwarder and I can successfully see the SC4S is properly communicating with Splunk. After that, I used Kiwi Syslog Message Generator from my windows 10 machine to send a syslog tcp message to the Centos 7 VM. Successful Output (TCP): However, if i sent a syslog udp message, the message was not successfully sent. As shown in the screenshot, the messages sent was zero after i pressed send. Unsuccessful Output (UDP): No new messages were shown in Splunk Web. 514 TCP and UDP is enabled in the firewall in Centos 7. I would like to request assistance about this issue. Thank you.                   ... View more