I have checked the following, permissions on the add-on and input file and verified its correct. did reload deploy-server - used btool and got the following result on the Indexer, c:\>splunk btool props list --debug | findstr iis C:\Program Files\Splunk\etc\system\default\props.conf [iis] C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-iis\default\props.conf [ms:iis:auto] ... View more

Based on this link - link text It seems to be an issue with the openssl version. ... View more

Before using the Splunk Default Certs, I had tried setting up SSL using self signed itself, as I have mentioned in my post. However, because that was giving me the same errors, I thought to make it simpler by using Splunk defaults. ... View more

Forwarder - outputs.conf [tcpout:group1] server = 192.168.14.30:9997 clientCert = C:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem sslPassword = password sslVerifyServerCert = true useClientSSLCompression = true sslVersions = *,-ssl2 server.conf [sslConfig] sslPassword = password sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem sslVersions = *,-ssl2 Indexer inputs.conf [splunktcp-ssl:9997] disabled = 0 [SSL] serverCert = C:\Program Files\Splunk\etc\auth\server.pem sslVersions = *,-ssl2 sslPassword = password requireClientCert = true server.conf [sslConfig] sslPassword = password sslRootCAPath = C:\Program Files\Splunk\etc\auth\cacert.pem sslVersions = *,-ssl2 I had initially started with self-signed certs but to make it less complicated, I am using Splunk default certs. Atleast once it starts working properly, I will move to self signed. ... View more

When I run the below command, C:\Windows\system32>splunk cmd openssl s_client -connect 192.168.14.30:9997 -tls1_2 the connection is successful but with the below errors, CONNECTED(000000F0) depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com verify error:num=19:self signed certificate in certificate chain 164:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:.\ssl\s3_pkt.c:1498:SSL alert number 40 164:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:.\ssl\s3_pkt.c:659: --- However, when I run the command provided by you, it also connects but with no error. I also tried to change the ssl version to - sslVersions = *,-ssl2 However, after doing that, I get this error, ERROR TcpInputProc - Error encountered for connection from src=192.168.14.10:50890. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol ... View more

Hi Harsh, I have configured with above with Splunk default CA. I was able to establish a successful comms from the forwarder to Indexer using /opt/splunk/bin/splunk cmd openssl s_client -connect <INDEXER>:<SSLPORT> -cert <PATH_OF_FORWARDER_CERTIFICATE> -CAfile <PATH_OF_ROOTCA_CERTIFICATE> However, I am still getting the exact same error. ... View more

Hi Harsh, Here are the details, C:\Windows\system32>splunk cmd openssl s_client -connect 192.168.14.1:9997 CONNECTED(000000F0) depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com verify error:num=19:self signed certificate in certificate chain --- Certificate chain 0 s:/CN=SplunkServerDefaultCert/O=SplunkUser i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com 1 s:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com --- Server certificate -----BEGIN CERTIFICATE----- MIIDMjCCAhoCCQDQdsr0LOW0rTANBgkqhkiG9w0BAQsFADB/MQswCQYDVQQGEwJV UzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoM BlNwbHVuazEXMBUGA1UEAwwOU3BsdW5rQ29tbW9uQ0ExITAfBgkqhkiG9w0BCQEW EnN1cHBvcnRAc3BsdW5rLmNvbTAeFw0xODA3MTgwMTU2NDhaFw0yMTA3MTcwMTU2 NDhaMDcxIDAeBgNVBAMMF1NwbHVua1NlcnZlckRlZmF1bHRDZXJ0MRMwEQYDVQQK DApTcGx1bmtVc2VyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2pd4 Dm5hIWqTwum95KLpNLyW6Bh0vLDWv8N11IZvvmBalr9Mxm4zrdgzeIv8mc26dyg0 Q4D9HyNKGzS1VYDHACohqFpwy6tIXA090TF4jX44vAAL1aAuamKKeHXfudectPTn PEhwr8hUsNox4WcN9zI9vlBnDFXuByCaA3dMXOsihB/1kFLna4PCBD/VgM8bsi9Q QWrFhuHWFuNmjTTCKGKeAB8mcoOkDNiSK5pKknMqh+SAfljgioYEdqGLhBUD+cTf d5feJCWh3G0Hy1+xuC0r8mskcrWGMF0NXLRziskxV4Y3MREfhbRESYpu1i7jTC2X Zd5VLaJuupGPsgyDFQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCJEj9Zm0E7MEAD 7c1klOS1ZqtPEEUNULiISohJQcTOG17cbl1BZlmLGCgHgmAXhuWC3xoYzwMa6Asm vrsBtccyDvtXoudssjmzcdR7bd13ply3voMCXtDMTyZcgGzQSTuQtyxnVmzeYfeu /+ZTP5bck87uKy4v5ElMnwXGx6hATfuaKDcJHPONwU++HBxfMW/mSNhLPQEV7kzT dcsgB9EFnIYxD7LAVaIAmjTaP5Us5IQ+x5Hw+3nAYW2EvY5WtUEnwRB5BmMPE6eQ Wh9RrdBxA8GVoodiFnjkEX/BqAo8akE3mECBcgEbg2zP20KWODTlq3ubfkdVbVSn IpdY5e63 -----END CERTIFICATE----- subject=/CN=SplunkServerDefaultCert/O=SplunkUser issuer=/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2387 bytes and written 441 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: zlib compression Expansion: zlib compression No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 219FFC9E3E943B40B0362FA4EDBCC763873F6FE0B1055F0AE35850A0B2378BE0 Session-ID-ctx: Master-Key: 76CA9533CD8C2C755EBD9C81EAF9A3967CEA54574FF557C63F0C68017612EA655FC58F56831E58F29C51575DB84C978C Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - f7 7d 00 e8 4e 4c 25 f9-a1 d8 fc 89 2b a7 e5 09 .}..NL%.....+... 0010 - 84 bf 4f ec 38 0a b1 d9-84 cf 15 04 16 ca f8 d5 ..O.8........... 0020 - 8d bc cf 45 fb 2f 26 9c-2f 23 ff 69 ef 02 33 77 ...E./&./#.i..3w 0030 - 52 56 b8 d0 98 6d c2 19-79 31 b0 5a 7c 80 56 8d RV...m..y1.Z|.V. 0040 - 42 1a be a0 2f a7 ef 83-8f 03 d2 75 be 8e a8 f0 B.../......u.... 0050 - 6d 5a d7 b1 db 6c 66 de-6b 5f 7d 49 0a 0e 5b 73 mZ...lf.k_}I..[s 0060 - f9 30 95 5c 55 c7 52 83-65 35 d5 fc 86 19 01 69 .0.\U.R.e5.....i 0070 - cb 8f c5 7c cc c5 3a 6d-f7 78 98 34 04 b5 66 58 ...|..:m.x.4..fX 0080 - 09 20 3b 97 45 0c 0d e3-ba 04 50 a8 57 b9 ef 6e . ;.E.....P.W..n 0090 - ac 6f 08 9b ae 85 f9 a0-68 7d 36 26 74 90 5d 1f .o......h}6&t.]. Compression: 1 (zlib compression) Start Time: 1536899254 Timeout : 300 (sec) Verify return code: 19 (self signed certificate in certificate chain) --- read:errno=0 On the forwarder Splunkd log, I get this message, DEBUG TcpOutputProc - channel not registered yet DEBUG TcpOutputProc - Connection not available. Waiting for connection and on indexer, INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL ... View more

So does that mean, we need to create client.pem on each universal forwarder ? ... View more

With that, I only got one result same as the first in your screenshot - Operation=create. I am suspecting someone ran splunk clean eventdata -index test_index on cli. Is there anyway to find user who executed this command ? Thanks. ... View more

You are right. There was a miscommunication with the client. These logs are actually from Meraki! Thanks for your assistance on this. ... View more

Yes, the add-on has been deployed literally everywhere, the S.H, indexer, heavy forwarder. The data is in this format now, <134>1 14245934.01624555 tfC_G_Af14 flows allow src=14.15.150.50 dst=152.56.5.51 mac=50:5E:55:53:82:30 protocol=tcp sport=3560 dport=56 Thats the Cisco PIX PFSS format (raw logging) set in Kiwi ... View more

Hi @FrankVI, Configured Kiwi log file format as raw logs In 1st attempt, removed host_segment=2 to check whether Cisco's add-on extracts the IP but it didnt work as expected. Resulting host value = syslog server IP. 2nd attempt, brought back host_segment=2, host IP extracted properly. However, the Cisco networks app is still not showing any data. it looks like transforms is still not getting applied. Its already using the sourcetype = syslog as shown above in monitoring stanza of in the inputs.conf ... View more