There  are five different hosts on our fleet on two different timezones with four sourcetypes on each. The problem is that the time that is being shown in Splunk Cloud isn't always the timestamp from the logs. They are different.  The hosts pass the data through an intermediate forwarder (universal forwarder running inside) which is in UTC.  There's also cases where one sourcetype from one host shows up/parses the correct time format but when they are coming from a different source, it doesn't.  I'll explain below: Five different hosts - host_A (MST), host_B (MST), host_C (UTC), host_D (UTC), host_E (UTC) Four different source types - src_W, src_X, src_Y, src_Z For host_A (MST) and host_B (MST), src_W is shown at the correct time. src_X and src_Y are not. For example - if src_X and src_Y have the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 7:xx:xx.  Between these two, src_Z only comes from host_A and the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx.  For host_C (UTC) - if src_W and src_X have the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_C doesn't have Y and Z.  For host_D (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_D doesn't have the other sourcetypes. For host_E (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_E doesn't have the other sourcetypes. For src_Z timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx - just like in host_A. Sorry this might seem to be very complicated and it is in MST and not PST like I said before. My Splunk Cloud instance is also set to MST.  Below is how the log formatting looks like: This is how log from src_W is:  eni=xx.yy.zz.aa client_ip=- - - [05/Feb/2023:17:46:53 -0700] ... ... .... This is how log from  src_X is: DEBUG 2023-02-06 00:49:22 ... ... ... This is how log from src_Y is:  INFO 2023-02-06 00:50:02 ... ... ... This is how log from src_Z is: qwertyui Sun Feb 5 04:40:39 2023: Thank you for the help! ... View more

Background: I am sending data to Splunk Cloud through an Intermediate Forwarder, which is a universal forwarder from multiple source instances (both in Pacific time and UTC) that do not support HEC and the correct TLS versions. As of now, this is the only way of sending the logs.  Sources (PT and UTC) > Intermediate Forwarder > Splunk Cloud Problem:  Some of the source instances are in Pacific time zone and the intermediate forwarder is in UTC. The logs that are coming from the instances that are in PT, are showing up in UTC time in Splunk Cloud. So Splunk Cloud is showing that the logs are from a time earlier than the they really got generated.  I should also say that the logs have a timestamp (PT - correct time) in them, always within the first 70 characters or so at most. That is the time I want them to be shown in.  Also changing the intermediate forwarder timezone to PT fixes the issue for the instances in PT but messes up the instances that are in UTC.  As a solution, I saw that timezones can be configured in the props.conf file, located in (after creation) /opt/splunkforwarder/etc/system/local directory.  Here is what it looks like for me:    [host::<some of the hosts>] TZ = US/Pacific   I have tried adding the following argument in the stanza (according to this post https://community.splunk.com/t5/Getting-Data-In/Universal-Forwarder-and-props-conf-and-transforms-conf/m-p/39732/highlight/true#M7401) but it is of no help.    force_local_processing = true    However, this just does not work. I can see logs in Splunk Cloud but they are in UTC.  I have also tried using <sourcetype> and source::<source> instead of [host::<myhost>] but it doesn't do anything.  Any other things that I can try? ... View more

I have Splunk UF 7.0.3 that I want to send logs from to Splunk Cloud.  However, the UF doesn't support httpout so I am using an intermediate forwarder.  Can someone give me the input and output files of the intermediate forwarder and the output file to send to the intermediate forwarder?    ... View more

I have a RHEL5 instance running  Universal Forwarder 7.0.3, currently sending logs to Splunk Enterprise. We are in the process of migration to Splunk Cloud. Splunk Cloud doesn't accept <TLS 1.2 and I can't use HEC from the host because the TLS version is 1.0.  As part of the solution, I came up with using an intermediate forwarder - this can forward the logs however, what I am getting is all hex characters.  Something like this: \x00\x8F
\x00\x00\x8Bo\xF5\x86\x84᜝h\xFCt5\xCB4T^\x9B\xBC\xE3c\xE6i\xD3\xA5\xCE/\x00\x00 \xC0,\xC00\xC0+\xC0/\xC0$\xC0(\xC0#\xC0'\x00\x9D\x00\x9C\x00<\xC0.\xC0-\xC0&\xC0%\x00\xFF
\x00\x00A\x00\x00\x00
\x00 At some point, I also saw "--splunk-cooked-mode-v3--" in the logs.  The inputs file of the for the intermediate forwarder is this: [splunktcp://<Source IP>:<Port>] index = <my index> disabled = false The output is just the standard HEC.  The version of the universal forwarder that I am using is 9.0.3 The universal forwarder version of the source cannot be updated to the latest one or any more than that since it is RHEL5.  How should I be able to see clean data and not hex ones?  ... View more

I am trying to install PIP on the splunk provided python - python3.7.  From what I can see, python is located on /opt/splunk/bin directory, so I am using the command: python3.7 get-pip.py --user after getting in that directory. However, I am getting errors, such as: ModuleNotFoundError: No module named 'distutils.command'   Is there already pip installed here? How    What do I do? As far as I know pip usually comes with python3 but I do not see it here. Without pip, it is extremely difficult to install any package so I am in a loop.    ... View more