Hi All,  We have concern raised by one of our application team as they could see incorrect data in their dashboard, When validated the same by looking into the source of the file where the splunk is reading it, we noticed that there is no actual data present in the log source. Problem:  Getting incorrect data ingested into Splunk in the status field value [13/Apr/2022:06:33:03 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2 [13/Apr/2022:04:30:01 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2 [12/Apr/2022:09:11:37 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2 Actual data present in the application server  Path:/var/mware/logs/xxx/localhost_access_log.2022-04-12.11.log [12/Apr/2022:11:10:26 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=0 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=0 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 Monitoring Stanza details: [monitor:///var/mware/logs/*/*localhost*] sourcetype = access_combined index = test disabled = 0 ignoreOlderThan = 1d blacklist=\.(gz)$ Splunkd.log : There is no significant ERROR|WARN|INFO related to this issue found. So could you please guide me what will be the reason why Splunk is ingesting an incorrect information when there is no actual data present in the source and also guide me how to troubleshoot this issue. ... View more

Hi All, One of our Cyber security person facing a strange issue while trying to access the data from the Splunk search portal. Initial level of troubleshooting the issue we found that Roles/Permission are not syncing but later we found that Roles/Permission are auto changing frequently. We could not find any ERROR/WARN in the splunkd.log, so not sure how to troubleshoot this issue Splunk version : 8.2  OS: Linux  Authentication mode: LDAP  Environment: Splunk distributed Production Environment.  Problem statement:  Roles/Permission are not syncing properly its getting auto changed frequently.  Kindly let me know what are steps we should follow to troubleshoot this type of issue. ... View more

Hi All, We wanted to do POC for our client and wanted to ingest open telemetry data logs and trace into splunk and I have following questions?  Is it possible to do them in Splunk Enterprise trail license? Or Do we need to buy Splunk Observability module to monitor the open telemetry data? Can we use universal forwarder to collect the logs and trace or do we need to have the Splunk OpenTelemetry Connector? Share the link or document to ingest the open telemetry data logs into splunk. ... View more

Hi All, Kindly let me know if there is any document or link which can provide me steps to  on-board the open telemetry data in to splunk Enterprise.   1) Do we need to buy the Splunk Observability cloud in-order to monitor/analysis the open telemetry data ? 2)  What are the steps or Procedure which we need to follow to on-board the open telemetry data in to Splunk? Kindly provide the link to access the document. 3) I had gone through this link but getting confused on what on the component which are need to perform this task.  https://docs.splunk.com/Observability/get-started/welcome.html#nav-Welcome-to-Splunk-Observability-Cloud  thanks in advance.  ... View more

Hi, One of our client interested to ingest the open telemetry data in to splunk and  we want to monitor Application performance and I could see Splunk APM module is used to perform this task.  When I was going through the Splunk Distribution open telemetry collector documents, I got the following doubts. 1) Do I need to buy any additional modules in-order to ingest the Open telemetry data in to splunk, as we have Splunk Enterprise Licensed version.   2)  Is it possible to ingest the open telemetry data without using the Splunk Observability Cloud module.  3) If it is possible to ingest open telemetry data into splunk than could you please provide the link to access the document. Thanks in advance.  ... View more