Same error. In essence, it doesn't recognise splunk_hec as a possible exporter. I'm on the latest version of Opentelemetry collector.  ... View more

This works fine (I used "form"  insted of "dashboard"  as a dashboard has many inputs). Thanks! Sz ... View more

I finally managed to make it work by manually assigning TZ to sourcetype in props.conf on HF.  Thank you @isoutamo ! ... View more

My app always uses UTC to put a timestamp. However, I always managed to use index time to display events correctly (ignoring app timestamp), but not in this case. As I said before, the logs are being displayed correctly from my lab system (the same app, the same timestamp set).  Weird.   ... View more

Do you mean "search and reporting" app? The only thing I can do is the restart Splunk cloud instance. Is it right?  ... View more

[root@xxx ~]# clock Fri 21 Jul 2023 16:15:09 BST -0.286435 seconds [root@xxx ~]# tail -n 10 /opt/videoipath/logs/backend/main.log 2023-07-21 15:15:12,326 backend_2023.2.5: INFO content 2023-07-21 15:16:48,011 backend_2023.2.5: INFO content  And below is how logs are being visible in GUI   ... View more

Would this be enough?  [SSL] _rcvbuf = 1572864 allowSslRenegotiation = true certLogMaxCacheEntries = 10000 certLogRepeatFrequency = 1d cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ecdhCurves = prime256v1, secp384r1, secp521r1 host = $decideOnStartup index = default logCertificateData = true sslQuietShutdown = false sslVersions = tls1.2 [batch:///opt/splunkforwarder/var/run/splunk/search_telemetry/*search_telemetry.json] _rcvbuf = 1572864 crcSalt = <SOURCE> host = $decideOnStartup index = _introspection log_on_completion = 0 move_policy = sinkhole sourcetype = search_telemetry [batch:///opt/splunkforwarder/var/spool/splunk] _rcvbuf = 1572864 crcSalt = <SOURCE> host = $decideOnStartup index = default move_policy = sinkhole [batch:///opt/splunkforwarder/var/spool/splunk/...stash_hec] _rcvbuf = 1572864 crcSalt = <SOURCE> host = $decideOnStartup index = default move_policy = sinkhole sourcetype = stash_hec [batch:///opt/splunkforwarder/var/spool/splunk/...stash_new] _rcvbuf = 1572864 crcSalt = <SOURCE> host = $decideOnStartup index = default move_policy = sinkhole queue = stashparsing sourcetype = stash_new time_before_close = 0 [batch:///opt/splunkforwarder/var/spool/splunk/tracker.log*] _rcvbuf = 1572864 host = $decideOnStartup index = _internal move_policy = sinkhole sourcetype = splunkd_latency_tracker [blacklist:/opt/splunkforwarder/etc/auth] _rcvbuf = 1572864 host = $decideOnStartup index = default [blacklist:/opt/splunkforwarder/etc/passwd] _rcvbuf = 1572864 host = $decideOnStartup index = default [fschange:/opt/splunkforwarder/etc] _rcvbuf = 1572864 delayInMills = 100 disabled = false filesPerDelay = 10 followLinks = false fullEvent = false hashMaxSize = -1 host = $decideOnStartup index = default pollPeriod = 600 recurse = true sendEventMaxSize = -1 signedaudit = true [http] _rcvbuf = 1572864 ackIdleCleanup = true allowSslCompression = true allowSslRenegotiation = true dedicatedIoThreads = 2 disabled = 1 enableSSL = 1 host = $decideOnStartup index = default maxSockets = 0 maxThreads = 0 port = 8088 sslVersions = *,-ssl2 useDeploymentServer = 0 [monitor:///opt/splunkforwarder/etc/splunk.version] _TCP_ROUTING = * _rcvbuf = 1572864 host = $decideOnStartup index = _internal sourcetype = splunk_version [monitor:///opt/splunkforwarder/var/log/splunk] _rcvbuf = 1572864 host = $decideOnStartup index = _internal [monitor:///opt/splunkforwarder/var/log/splunk/configuration_change.log] _rcvbuf = 1572864 host = $decideOnStartup index = _configtracker [monitor:///opt/splunkforwarder/var/log/splunk/license_usage_summary.log] _rcvbuf = 1572864 host = $decideOnStartup index = _telemetry [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log] _TCP_ROUTING = * _rcvbuf = 1572864 host = $decideOnStartup index = _internal [monitor:///opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log*] _rcvbuf = 1572864 host = $decideOnStartup index = _telemetry sourcetype = splunk_cloud_telemetry [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log] _TCP_ROUTING = * _rcvbuf = 1572864 host = $decideOnStartup index = _internal [monitor:///opt/splunkforwarder/var/log/watchdog/watchdog.log*] _rcvbuf = 1572864 host = $decideOnStartup index = _internal [monitor:///opt/videoipath/logs/backend/main.log] _rcvbuf = 1572864 disabled = false host = $decideOnStartup index = discol2 sourcetype = main_demo [script] _rcvbuf = 1572864 host = $decideOnStartup index = default interval = 60.0 start_by_shell = true [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip host = $decideOnStartup index = default route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue [tcp] _rcvbuf = 1572864 acceptFrom = * connection_host = dns host = $decideOnStartup index = default [udp] _rcvbuf = 1572864 connection_host = ip host = $decideOnStartup index = default ... View more

Hi Ismo,  Below is TZ set on UF server. Interestingly I set the same time zone on my local test server and after changing timestamp settings to "current time" the logs are displaying in time. However, it's a simple UF->Cloud setup as opposed to productional one (through HF). timedatectl Local time: Fri 2023-07-21 13:27:32 BST Universal time: Fri 2023-07-21 12:27:32 UTC RTC time: Fri 2023-07-21 12:27:32 Time zone: Europe/London (BST, +0100) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: yes Last DST change: DST began at Sun 2023-03-26 00:59:59 GMT Sun 2023-03-26 02:00:00 BST Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2023-10-29 01:59:59 BST Sun 2023-10-29 01:00:00 GMT Not sure how to grab log event on UF (this sounds like from Windows world). regards, Sz ... View more

@bowesmana You nail it! I had to play with regex a little bit, but I managed to achieve what I wanted.  Thanks a lot.  ... View more

Dear @bowesmana  Apologies for not being entirely clear in previous messages. No wonder you got confused. Let me clarify this again. I got logs: Splunk_timestamp EventA time_used [249.34ms] Splunk_timestamp EventB time_used [246.23ms] Splunk_timestamp EventC time_used [323.11ms] Splunk_timestamp EventA time_used [1445.12ms]  I managed to extract time_used value per event.  I would like to have a total sum of time_used from all events per hour.  The next step would be having percentage of time_used total number/number of ms in day i.e. (3 600 000 ms - 1000ms*60*60).  My query was a crap, so I'm not attaching it here.  ... View more

Hi,   I don't take into account the timestamp at the beginning of every line. Relying on Splunk-added timestamp is okay.  716906.314 - 716920.165  I'm struggling with an accumulation of time  in ms at the end of every log line like: 24.051   3124.593   and divide it into hours (Splunk timestampt)  ... View more

I've already given it a go (not in props.conf, but in the sourcetype edit tab in GUI - I'm using cloud premise).  It doesn't break lines correctly.    ... View more

The thing is that the network isn't being managed by me. I was just told to take advantage of HF (UF doesn't have access to the internet) and make everything up and running. But according to the customer, who is managing the network, there is nothing that could block communication between HF and UF. ... View more

My bad. It's typo. The second output is from HF.  Anyway I'm pasting it here again:  [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip host = $decideOnStartup index = default route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue [splunktcp://9997] _rcvbuf = 1572864 connection_host = 10.24.118.91 disabled = false host = $decideOnStartup index = discol ... View more

From UF [syslog] maxEventSize = 1024 priority = <13> type = udp [tcpout] ackTimeoutOnShutdown = 30 autoLBFrequency = 30 autoLBVolume = 0 blockOnCloning = true blockWarnThreshold = 100 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256 compressed = false connectionTTL = 0 connectionTimeout = 20 defaultGroup = default-autolb-group disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 ecdhCurves = prime256v1, secp384r1, secp521r1 forceTimebasedAutoLB = false forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry) forwardedindex.filter.disable = false heartbeatFrequency = 30 indexAndForward = false maxConnectionsPerIndexer = 2 maxFailuresPerInterval = 2 maxQueueSize = auto readTimeout = 300 secsInFailureInterval = 1 sendCookedData = true sslQuietShutdown = false sslVersions = tls1.2 tcpSendBufSz = 0 useACK = false useClientSSLCompression = true writeTimeout = 300 [tcpout-server://172.23.11.216:9997] [tcpout:default-autolb-group] disabled = false server = 172.23.11.216:9997 sslPassword = password From UF [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip host = $decideOnStartup index = default route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue [splunktcp://9997] _rcvbuf = 1572864 connection_host = 10.24.118.91 disabled = false host = $decideOnStartup index = discol I've only just added connection_host setup , but it didn't make any difference.  Logfile: 11-17-2021 09:31:50.643 +0000 WARN TcpOutputProc [18893 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=172.23.11.216 inside output group default-autolb-group from host_src=ldcrapnvvip10 has been blocked for blocked_seconds=47000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 11-17-2021 09:32:26.647 +0000 ERROR TcpOutputFd [18894 TcpOutEloop] - Read error. Connection reset by peer 11-17-2021 09:32:26.648 +0000 ERROR TcpOutputFd [18894 TcpOutEloop] - Read error. Connection reset by peer 11-17-2021 09:32:26.648 +0000 WARN AutoLoadBalancedConnectionStrategy [18894 TcpOutEloop] - Applying quarantine to ip=172.23.11.216 port=9997 _numberOfFailures=2   ... View more

inputs.conf file from $SPLUNK_HOME/etc/apps/search/local [splunktcp://9997] disabled = false Forget my ignorance, but should I add this stanza to $SPLUNK_HOME/etc/system/local/inputs.conf file as well?    ... View more

@PickleRick I double-checked TLS/SSL configuration on both sides. Looks like default setting and it's the same. I've check memory utilization on HF and it's quite high:  96% memory is consumed. Could it be a culprit?  ... View more

@PickleRick ACK/RST packet turns up after 3-way handshake.  I caught some TLSv1 packets. Looks like TLS hello packet from UF towards HF, so your guess related to TLS mismatch can be right.  ... View more