All Topics

Top

All Topics

Hi, Everyone!   Thinking about how Smart Agent integrates with your CI/CD pipelines? Is agent management encouraged for existing CI/CD pipelines? Um..., yes! See the details here, and please sha... See more...
Hi, Everyone!   Thinking about how Smart Agent integrates with your CI/CD pipelines? Is agent management encouraged for existing CI/CD pipelines? Um..., yes! See the details here, and please share your questions and impressions below: Smart Agent FAQ | Tooling Pipeline Guidelines What do you think? Our team would love to hear your thoughts, including how we can add to and improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate a quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, Everyone!   Starting to think about your agent management strategy?  Check out topical questions to spur your planning and inspire more questions: Smart Agent FAQ | Strategy How does Smart ... See more...
Hi, Everyone!   Starting to think about your agent management strategy?  Check out topical questions to spur your planning and inspire more questions: Smart Agent FAQ | Strategy How does Smart Agent manage existing agents—or new planned ones? What if there are hundreds, or more? How will it really work for your day-to-day? What do you think? Our team would love to hear your thoughts, including how we can add to and improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate this quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, Everyone!   What are Smart Agent's requirements?  Here's a key question before getting started: What are the requirements?  Smart Agent FAQ | Requirements What do you think? We've starte... See more...
Hi, Everyone!   What are Smart Agent's requirements?  Here's a key question before getting started: What are the requirements?  Smart Agent FAQ | Requirements What do you think? We've started with the requirements questions we knew you'd want, plus questions others have already asked. Our team would love to hear your thoughts, including how we can add to and improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate this quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, Everyone!   What about environments supported with Smart Agent?  Find out what environments and features are supported as what may be coming down the line. Post your questions and we will be su... See more...
Hi, Everyone!   What about environments supported with Smart Agent?  Find out what environments and features are supported as what may be coming down the line. Post your questions and we will be sure to address them. The future of Smart Agent depends on your needs! Smart Agent FAQ | Supported Environments What do you think? Our team would love to hear your thoughts, including how we can add to and improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate this quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, Everyone!   Concerned about costs or support with Smart Agent?  Spoiler alert: You don't need to buy additional licenses to use Smart Agent. Check out the other most frequently asked about this... See more...
Hi, Everyone!   Concerned about costs or support with Smart Agent?  Spoiler alert: You don't need to buy additional licenses to use Smart Agent. Check out the other most frequently asked about this here:  Smart Agent FAQ | Licenses and Packages  What do you think? Our team would love to hear your thoughts, including how we can add to and improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate this quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, Everyone!   Get to know some of the basics around using Smart Agent to simplify agent management tasks, such as it's value and what specific features are available.  Smart Agent FAQ | Simplifi... See more...
Hi, Everyone!   Get to know some of the basics around using Smart Agent to simplify agent management tasks, such as it's value and what specific features are available.  Smart Agent FAQ | Simplified Agent Management Basics There, find out how we define smart agent management, installation requirements and details, a high-level outline of value in this v23.11 release,  and more...  What do you think? Our team would love to hear your thoughts, and how we can improve the FAQ Please do share your impressions, considerations, and questions below. Our Smart Agent FAQ has a lot of information about Smart Agent and related features. We thought you might appreciate this quick way to get to the topics that most interest you, paired with a place to ask questions and enlarge on your take... 
Hi, I have noticed over the last 4 days I had an increased number of Search Bundle replication errors: 12-21-2023 09:50:12.604 +0000 WARN ConfReplicationThread [9209 ConfReplicationThread] - Error ... See more...
Hi, I have noticed over the last 4 days I had an increased number of Search Bundle replication errors: 12-21-2023 09:50:12.604 +0000 WARN ConfReplicationThread [9209 ConfReplicationThread] - Error pushing configurations to captain=https://searchHeadCaptain:8089, consecutiveErrors=1 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=16ed9160640170315673324237791a4cfe256d59; current_baseline_op_id=cd93950208af34df00957e721b87128d3629d2d1" These occur in groups every 4 hours. I have also seen CPU spikes on the Search Heads that started occuring at the same time and also every 4 hours. Further investigation has shown that the following events from conf.log have also been occuring at the same time every 4 hours { [-]    component: ConfOp    data: { [-]      applied_at: 1703264397      asset_id: 220d8bbce6d790850cda3980c5784c62b1a9f9ff      asset_uri: [ [+]      ]      from_repo: https://searchHeadCaptain:8089      op_id: 102aa206f930da5eef0d47163b354c61254566c5      optype: 2      optype_desc: WRITE_STANZA      payload: { [-]        alias: Risk        metadata: { [-]          permissions: { [-]          }        }        value: ***TRANSIENT***://6613      }      payload_extra: ***ALLOW_SKIP_ON_WRITE***      status: applied      task: pullFrom      to_repo: https://searchHeadPeer.com:8089      to_repo_change_count: 20214    }    datetime: 12-22-2023 16:59:57.097 +0000    log_level: INFO } Does anyone know what these events mean and how I can find out what is causing them? Bundle replication errors:   conf.log events:   CPU spikes:  
I am running the current search using the network toolkit but will not show the hostname field from the csv, do I need to do another inputlookup at the end of the search. | inputlookup iphost.csv |... See more...
I am running the current search using the network toolkit but will not show the hostname field from the csv, do I need to do another inputlookup at the end of the search. | inputlookup iphost.csv | search src_ipV4=* hostname=* | rename src_ipV4 as host | stats values(host) as host | mvexpand host | map maxsearches=50 search="| ping host=$host$ count=1 | eval dest=if(isnull(dest),host,dest) | fields host dest received" | table host dest received hostname
We have data coming in that we need to alert on, however because of the formatting of the data, this is very hard to do.   The data is coming in as key value pairs but the values are not encapsulated... See more...
We have data coming in that we need to alert on, however because of the formatting of the data, this is very hard to do.   The data is coming in as key value pairs but the values are not encapsulated in quotes and is being truncated.  For example _Raw - filepath=c:\program files\abc123\ What we end up getting is Parsed - filepath=c:\program Everything after the space is ignored. If I wanted to find all occurrences where the path was c:\program files\abc123, I can't. We are sending the data via syslog to the splunk servers Thanks in advance!      
Is it standard for the Splunk server itself to be over 50% of the daily indexing total? In our production environment, we are starting run over the daily and simply because of the splunk server itsel... See more...
Is it standard for the Splunk server itself to be over 50% of the daily indexing total? In our production environment, we are starting run over the daily and simply because of the splunk server itself. I understand its what does the heavy lifting, but its hard to base how much licensing you may need when you dont know how to gauge what the server will utilize    
Hi, How to add MSAL4J.jar to DB_Connect. I am getting error: Failed to load MSAL4J Java library for performing ActiveDirectoryServicePrincipal authentication.
Hi all, We need to add a couple dozen new search head peers to search head deployer, as well as adding a couple dozen indexers to a cluster master and would like to script this implementation.  I n... See more...
Hi all, We need to add a couple dozen new search head peers to search head deployer, as well as adding a couple dozen indexers to a cluster master and would like to script this implementation.  I need to know what configuration files need to be modified to join these new search head peers and indexer to the existing Splunk environment.  We are plan on running an Ansible script for this implementation project.  /Paul
Hello Experts, I'm facing challenge where I need to automatically load data from Python script results into a metric index in Splunk. Is it possible? I'd appreciate any guidance or examples how to... See more...
Hello Experts, I'm facing challenge where I need to automatically load data from Python script results into a metric index in Splunk. Is it possible? I'd appreciate any guidance or examples how to achieve this? Thanks
In October, v23.10.x enhancements included Cisco Cloud Observability,  SaaS Controller and Agent enhancements—including SAP Agent updates, and On-premises Controller  upgrades. In November, v23.1... See more...
In October, v23.10.x enhancements included Cisco Cloud Observability,  SaaS Controller and Agent enhancements—including SAP Agent updates, and On-premises Controller  upgrades. In November, v23.11 enhancements included Agent Management features, including Smart Agent and the Smart Agent CLI. Product name change announcements  As of November 27, 2023, the Cisco Full-Stack Observability Platform is now the Cisco Observability Platform and Cloud Native Application Observability is now Cisco Cloud Observability powered by the Cisco Observability Platform. These name changes better align our products with the Cisco portfolio and with our business strategy.  WATCH THIS PAGE FOR UPDATES — Click the Options menu above right, then Subscribe Want to receive all monthly Product Updates? Click here, then subscribe to the series In this article…  What new product enhancements were released October-November 2023? Cisco Full-Stack Observability | Cisco Cloud Observability | Agents | SAP | SaaS Controller | On-premises Controller | Accounts Where can I find additional information about product enhancements?  Resolved Issues Advisories and Notices Essentials Download components  | Get started upgrading AppDynamics components for any release | Product Announcements, Alerts, and Hot Fixes | Open source extensions | License entitlements and restrictions What new product enhancements were released in October and November 2023? TIP | This article provides product enhancement highlights, organized by product. Each product section links to the corresponding Release Notes page. When available, links to the specific release version are also included.  Cisco Observability Platform enhancement highlights Formerly Cisco Full-Stack Observability Platform NOTE | There was no Cisco Observability Platform release during this period. Below, see module enhancement highlights. For a complete list of enhancements, see the  v23.10.0 Module Enhancements Release Notes and the  v23.11.0 Module Enhancements Release Notes. For Developer Support, see the 23.10.27 Cisco Observability Platform Release Notes. Extended Cisco Cloud Observability Modules  Application Resource Optimizer: Override blockers when configuring the optimization of your workloads for testing purposes. Use Cost Insights to set budgets and monitor costs for infrastructure resources grouped by teams. Create HTTP alerts on Cisco Secure Application Back to TOC | To Essentials   Cisco Cloud Observability enhancement highlights Formerly Cloud Native Application Observability prior to November 27, 2023 NOTE | See the Cisco Cloud Observability v23.10 Release Notes page for a complete list of enhancements in October 2023. There was no release in November 2023. Alerting GA 23.10.27 Health Rules have several new features, including:  Create health rule conditions for percentile values on a histogram metric—a feature that enhances alert generation accuracy by reducing noise. More in the Release Notes.  By establishing health rule conditions for events, you can also receive alerts for abnormal events. Violating Events Chart information includes the number of events during a set time, trend, and deviation from threshold value.  When applying filters for entity types and metrics, values of available filter attributes are now automatically suggested.  Purged Time to Live (TTL) for each entity documented Cisco AppDynamics has documented the purge time-to-live (TTL) for each entity. For more information, see Kubernetes Entities. For other domains, see the entity-centric page documentation for your specific entity.  Cloud Infrastructure and Troubleshooting   Kubernetes and App Service Monitoring  Application Performance Monitoring with OpenTelemetry  Cloud infrastructure troubleshooting   Cisco Cloud Observability now supports monitoring:  Amazon Apache Managed Flink Application  Amazon Athena  AWS CodeBuild  Amazon Cognito  Amazon Elastic Container Registry  AWS Glue  Amazon Simple Notification Service  GCP Cloud Run  GCP Cloud SQL  GCP Load Balancers Kubernetes and App Service Monitoring  The Orchestration Client was renamed to the Cisco AppDynamics Smart Agent. As part of this change, the fso-agent-mgmt-client section of the operator-values.yaml file was renamed to appdynamics-smartagent. For any upgrade from 23.6.0 to >=23.9.0, ensure that the Smart Agent values are updated in the operators-values.yaml file. See Upgrade or Uninstall Kubernetes and App Service Monitoring. Use new Kubernetes predefined health rule K8s CronJob Health Rule to identify failed cronjobs. You can now use Enable Additional Configurations section at Configure > Kubernetes and App Services to generate the configuration file with selected collectors and operating systems.     When you download the configuration files for operator and the required collectors, the configuration file includes the inline documentation of the additional settings. See Install Kubernetes and App Service Monitoring. Grafana plugin  The 23.9 version of the Grafana plugin includes the Include All option toggle. App Root Cause Analysis using Anomaly Detection You can now view Pod readiness and liveness probe information can on the Properties panel of Pods, Workloads, Clusters, and Namespaces. Logs Cisco Cloud Observability now supports log collection from additional sources:  Amazon Elastic Load Balancing (Amazon ELB) service logs (includes ALB, CLB, NLB)  Amazon Virtual Private Cloud (Amazon VPC) service logs  Applications running on Amazon Elastic Container Service (Amazon ECS)   Applications running on AWS Fargate   Applications running on non-Kubernetes® Linux hosts, such as Amazon Elastic Compute Cloud (EC2) and bare-metal Linux machines    You can now deploy the Log Collector on non-Kubernetes hosts, such as bare metal EC2s. Hosts must be running on Linux. In this deployment, the Log Collector sends logs from the host directly to the Cisco AppDynamics common ingestion service (CIS). The new Enable Additional Configurations option generates a collectors-values.yaml file for you, simplifying the Log Collector’s deployment onto your clusters. See Configure the Log Collector. A new interface, Configure > Log Processing, allows you to specify server-side log parsing rules to extract fields from incoming log messages at the time of ingestion. By saving these rules on the server, you can standardize field names, improve search performance, and apply the same rules to new log sources.  Data Masking with the Rule scope Menu. In Configure > Data Security, options to specify the scope of a data masking rule have expanded, allowing you to select a log attribute from the Rule scope pull-down menu, and specifying a value for that attribute.   Masking rules you created with the logFormat parameter still work, but are no longer editable. We recommend that you delete your existing data masking rules and recreate them using the Rule scope menu. See Mask Sensitive Data. The Relevant Fields panel on the Logs page now displays all available relevant fields with distinct values, count, and percentage—expanded from a hardcoded list of 5 fields. This panel provides the fastest and most convenient way to filter your view of log messages. See Troubleshoot with Logs.  Service time investigations  In the time range selector, you can now select a default time, a recently used time range, or a custom time where you can select from absolute or relative time. See Understand the Observe UI.  Business Transactions 23.10.27  Now, only users with appropriate permissions can “favorite” Business Transactions. For users with the Observer role, the favorites and shortcut options will be hidden. See Business Transactions, Favorites.  With Business Transactions, you can now configure and visualize the Revenue Loss metric  to correlate performance issues to business impacts and perform segment analysis. See Business Transactions.  You can now filter based on specific metrics. A new column for metrics appears in the list view depending on the filter. See Filters.  The UI theme has been updated to include new features such as updated colors, flowmap capabilities (see all entity pages), and light/dark mode options.  Troubleshooting documentation has been updated for users who cannot collect some traces due to their size being greater than the allowed limit. See Troubleshoot Application Performance Monitoring with OpenTelemetry. Spectro Cloud® Palette  The following Cisco Cloud Observability monitoring solutions can now be installed and/or configured using add-on packs in the Spectro Cloud® Palette user interface:  Kubernetes and App Service Monitoring  Application Performance Monitoring  Log Collection  Events Collection  This alternate installation process does not require manually accessing the Kubernetes cluster or using Helm chart commands.  Back to TOC | To Essentials   Agent enhancement highlights NOTE | See the AppDynamics SaaS v23.10 and AppDynamics SaaS v23.11 for the complete October and November 2023 agent enhancement details.  ABAP Agent GA 23.11.0  November 31, 2023  ABAP Agent now supports End User Monitoring (EUM) for SAP GUI sessions. See SAP GUI End User Monitoring. For additional release details, see Release Notes v23.11.0: SAP-319 (ASM-1325) Deferred Error Reporting SAP-323 (ASM-1345) Custom Naming Logic for Backends SAP-363 (ASM-1334): S/4 HANA 2022 FPS02: HTTP SKK Update and TLS 1.3/1.3 support SAP-363 (ASM-1334): S/4 HANA 2022 FPS02 Compatibility SAP-371 (ASM-1323): Event Limit Parametrization Various Monitoring improvements Analytics Agent  GA 23.10.0  October 31, 2023 Upgrades to third-party component Azul JRE to v8.72(8u382)  Log Analytics support for the following operating systems:  IBM AIX (Advanced Interactive eXecutive) 7.2.x  HP-UX (Unix) 11.31.x  See Collect Log Analytics Data.  GA 23.11.0  November 27, 2023 Third-party components dom4j, jetter-server, netty-all, and org.json were updated. C/C++ SDK GA 23.11.0  November 17, 2023 Support for: Normal Average Response Time TLS 1.2 and TLS 1.3 for communication with the Controller and Analytics Server. Cluster Agent GA 23.11.0  November 29, 2023 New option allows you to retain the instrumented configuration for all successful deployments during an upgrade or re-instrumentation. See Auto-Instrumentation Configuration. Database Agent GA 23.11.0  November 29, 2023 You can configure policies for Microsoft SQL Server database events: AG_LISTENER_IP_NOT_ONLINE AG_REPLICA_DISCONNECTED AG_REPLICA_NOT_SYNCHRONIZING See Microsoft SQL Server Database Events Reference and Database Events Reference. IBM Integration Bus Agent GA 23.11.0  November 7, 2023 Now, you can select IIB Agent when creating a match rule, which gives visibility and context for non-performant IIB business transactions. See Custom Match Rules. you can add HTTP parameters while configuring data collectors, providing HTTP header and payload information for IIB business transactions. See Data Collectors. iOS Agent Support for the iOS SDK installation using Swift Package Manager for iOS Agents v23.10.1 or higher. See Install the iOS SDK.  Java Agent GA 23.11.0  November 30, 2023   You can specify the Controller keystore filename and password in the JVM startup script. See Java Agent Configuration Properties and Enable SSL between the Java Agent and the Controller. There is now support for WebLogic EUM Automatic Injection (JSPs). See Automatic Injection of the JavaScript Agent. Machine Agent GA 23.11.0  November 30, 2023 Support for HP-UX. See Machine Agent Requirements and Supported Environments. Support for OpenShift tags to differentiate between master and worked nodes. See Server Tagging. PHP Agent GA 23.11.0  November 29, 2023 Support for Alpine Linux. See PHP Supported Environments. Upgrades to following third-party components util-linux and ZeroMQ .NET Agent GA 23.11.0  November 30, 2023 Code optimizations, performance improvements, and third-party library upgrades.    Back to TOC | To Essentials   SaaS Controller enhancement highlights NOTES |See the AppDynamics v23.10 SaaS Controller Release Notes page for the complete October 2023 enhancements. No SaaS Controller enhancements were released in November. Cisco Secure Application GA 23.10.0 October 31, 2023 This release includes support for OpenTelemetry. See Cisco Secure Application for OpenTelemetry. Anomaly Detection GA 23.11.0 November 30, 2023 On the Suspected Cause details page, the Top Deviating Metrics timeline now displays the evaluation period of an anomaly for precise identification of the time when the issue started. See Troubleshooting Anomalies. Agent Management Enhancement GA 23.11.0 November 30, 2023 Introducing Smart Agent, which allows you to use the Controller UI to perform bulk agent operations—install, upgrade, or rollback.  Agent Management also provides an auto-attach feature for detecting and starting supported AppDynamics agents without modifying the start configuration of the applications. Smart Agent CLI provides buildtime workflows toautomate the installation, upgrade and uninstall of supported AppDynamics agents (including Smart Agent). Back to TOC | To Essentials   On-prem enhancement highlights NOTE |See the On-premises Platform Release Notes page for the complete October enhancements. No enhancements were released in November. Enterprise Console GA 23.10.0  October 12 31, 2023  This release includes Controller 23.7.3.  Back to TOC | To Essentials Where can I find additional information about product enhancements?  In Documentation, each product category has a Release Notes page where enhancements are described in detail on an ongoing basis. Links to the most recent versions are:  Cisco Observability Platform  Cisco Cloud Observability  AppDynamics SaaS Documentation (latest)  AppDynamics On-premises Documentation  Accounts Administration (Administration Tasks) SAP Monitoring using AppDynamics  Back to TOC | To Essentials Resolved issues DID YOU KNOW? You can find ongoing lists of Resolved Issues on each Release Notes page by version. Sort the list on each page by headings, including key, product, severity, or affected version(s). Find Resolved Issues by Product here:  Cisco Observability Platform  Cisco Cloud Observability  AppDynamics SaaS Documentation (latest)  AppDynamics On-premises Documentation Accounts Administration (Administration Tasks) SAP Monitoring using AppDynamics  Back to TOC | To Essentials Advisories and Notifications Upcoming End of Support for Cluster Collectors <23.10 and required upgrade for continued Kubernetes monitoring   Cluster Collectors <23.10 are deprecated with support ending January 30, 2024. After this date, monitoring Kubernetes entities via the relationship pane will require an upgrade to Cluster Collector version >=23.10.  Essentials ADVISORY | Customers are advised to check backward compatibility in the Agent and Controller Compatibility documentation. Download Essential Components (Agents, Enterprise Console, Controller (on-prem), Events Service, EUM Components) Download Additional Components (SDKs, Plugins, etc.) How do I get started upgrading my AppDynamics components for any release? Product Announcements, Alerts, and Hot Fixes Open Source Extensions License Entitlements and Restrictions CAN'T FIND WHAT YOU'RE LOOKING FOR? NEED ASSISTANCE? Connect in the Forums
Hi everyone, I am in trouble. I need help. We are performing an UPGRADE of splunk ITSI. Following the upgrade path of ITSI, we are now handling the following. 4.9.x → 4.11.x → 4.13.x → 4.15.x Tr... See more...
Hi everyone, I am in trouble. I need help. We are performing an UPGRADE of splunk ITSI. Following the upgrade path of ITSI, we are now handling the following. 4.9.x → 4.11.x → 4.13.x → 4.15.x Trouble is occurring in the following cases 4.9.6 → 4.11.6 The server configuration is a cluster. The Splunk version is as follows. Search head:Splunk 9.1.2 indexer:Splunk 9.1.2 api(HF):Splunk 9.1.2   Migration logs at the time the trouble occurred are as follows. -------------------------------------------------------- 2023/12/22 14:45:20.640 2023-12-22 14:45:20,640+0900 process:2531 thread:MainThread ERROR [itsi.migration] [itsi_migration:4543] [run_migration] Migration from 4.9.2 to 4.10.0 did not finish successfully. host = logmng-st-splunk_srch01source = /opt/splunk/var/log/splunk/itsi_migration_queue.logsourcetype = itsi_internal_log 2023/12/22 14:45:20.636 2023-12-22 14:45:20,636+0900 process:2531 thread:MainThread ERROR [itsi.migration] [__init__:1433] [exception] 4.9.2 to 4.10.0: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_entity_management_rules?fields=object_type; [{'type': 'ERROR', 'code': None, 'text': 'An object with name=itsi_entity_management_rules does not exist'}] Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-ITOA/lib/migration/migration.py", line 310, in run if not command.execute(): File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/itsi_migration.py", line 249, in execute backup.execute() File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 1244, in execute self.backup() File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 973, in backup raise e File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 942, in backup object_types = self._get_object_type_from_collection(collection) File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 601, in _get_object_type_from_collection rsp, content = simpleRequest(location, sessionKey=self.session_key, raiseAllErrors=False, getargs=getargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 669, in simpleRequest raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body)) splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_entity_management_rules?fields=object_type; [{'type': 'ERROR', 'code': None, 'text': 'An object with name=itsi_entity_management_rules does not exist'}] host = logmng-st-splunk_srch01source = /opt/splunk/var/log/splunk/itsi_migration_queue.logsourcetype = itsi_internal_log   2023/12/22 14:45:20.635 2023-12-22 14:45:20,635+0900 process:2531 thread:MainThread ERROR [itsi.migration] [__init__:1433] [exception] 4.9.2 to 4.10.0: BackupRestore: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_entity_management_rules?fields=object_type; [{'type': 'ERROR', 'code': None, 'text': 'An object with name=itsi_entity_management_rules does not exist'}] Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 1244, in execute self.backup() File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 973, in backup raise e File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 942, in backup object_types = self._get_object_type_from_collection(collection) File "/opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/kvstore_backup_restore.py", line 601, in _get_object_type_from_collection rsp, content = simpleRequest(location, sessionKey=self.session_key, raiseAllErrors=False, getargs=getargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 669, in simpleRequest raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body)) splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_entity_management_rules?fields=object_type; [{'type': 'ERROR', 'code': None, 'text': 'An object with name=itsi_entity_management_rules does not exist'}] host = logmng-st-splunk_srch01source = /opt/splunk/var/log/splunk/itsi_migration_queue.logsourcetype = itsi_internal_log -------------------------------------------------------- How do you deal with errors? Any help would be good! thanks, shinsuke
Hi, We initially deployed a heavy forwarder on-prem to collect data from our passive devices (syslogs, security devices, etc) however per talking with a splunk represent he recommended to have a s... See more...
Hi, We initially deployed a heavy forwarder on-prem to collect data from our passive devices (syslogs, security devices, etc) however per talking with a splunk represent he recommended to have a splunk connect for syslog to collect the data. Per him Syslog connect is the recommended method of collection for passive devices and also helps with parsing/normalization of the data when it goes to our Enterprise Security. Can both HF and SC4S be in the server ? If yes how will that work? Can SC4S direct data to the cloud indexer? And for future, do we just go for SC4S instead on the HF on-prem for the passive devices?  Thank you
Hi there. I would like to know about Splunk Health engine, Enterprise 8.2.12, 3 SHC,     HOW it considers a savedsearch a Lagged search? Based on same previous 24h search runs and doing an a... See more...
Hi there. I would like to know about Splunk Health engine, Enterprise 8.2.12, 3 SHC,     HOW it considers a savedsearch a Lagged search? Based on same previous 24h search runs and doing an average running times? Since we have many many heavy searches that end up also in 10/15m WHY, sometimes, i found in Skipped search monitor a 100% of skipped search (1 from 1, when we have hundreds of scheduled searches)? WHILE, searching the scheduler log, i found something like 70.000 success / 68 skipped (scheduled every minute or every two, concurrency is a factor i calculate and there's no problem) in last 24h ? WHY 100%? Is it a bug? I also search for a single scheduled search per day savedsearches, but all (few) are in "success" status When those strange things occur, sometimes, restarting the cluster, make health monitor to reset without warnings!!! Other times, in reverse, restarting the cluster make a clean health monitor to start giving warnings from point 1 & 2 ... strange behaviour!!! Thanks.
Common questions and answers about simplified Agent Management with Smart Agent  This FAQ articulates some of the most common questions and answers about simplified agent management with the Smart ... See more...
Common questions and answers about simplified Agent Management with Smart Agent  This FAQ articulates some of the most common questions and answers about simplified agent management with the Smart Agent and the enhanced user interface, along with the new CLI. In this article...  Simplified agent management basics  Licenses and packages  Supported Environments  Requirements  Strategy Tooling Pipeline Guidelines  UI Questions  Installation/Getting Started  Custom configuration files, monitors, and extensions  Additional resources Updated 1/24/24   Simplified agent management basics  FORUM LINK | Join the discussion on this topic What is simplified agent management?  Simplified agent management enables you to automate agent management operations such as installations, upgrades, and rollbacks at scale. This is achieved using the Smart Agent that facilitates and orchestrates agent operations by using the existing and enhanced Controller UI and/or command line interface (CLI). This applies to both SaaS and OnPrem customers alike.   The Smart Agent can manage the supported agents when the Smart Agent is installed on the same machine as a supported agent. You can use the enhanced Controller UI to view all the installed agents with the inventory details. The Smart Agent inventory is also displayed along with other agent inventories. (See below for more details.) You also have access to a CLI for advanced management controls.    Is the Smart Agent UI part of the current SaaS and On-prem UI, or will a separate installation be needed?   As of Controller v23.11, the existing Agent Management screen has been modified to include new information regarding Smart Agent, whether agents are being managed, their status, and more. To access this new Smart Agent UI, you will need to be on v23.11 at minimum.    What is the real value of this UI release? Will there be more than what we already have today?  This release offers significant standalone value. However, using Smart Agent enhances this value by providing the following features:  Simplified Upgrade Management  Enables you to choose the version to upgrade, download it from our repository, and make the necessary configuration changes all at the push of a button.  Upgrade Awareness  Effortlessly keep track of available upgrades.   These upgrades are visible on the right side of the screenshot, with distinct icons indicating their status—whether they're out of date, have updates ready for upgrade, are running the latest version, or have an unknown status.  Upgrade Rollback  Offers the ability to revert an upgrade if needed.  Progress Tracking  Provides a Tasks in Progress feature to keep you updated with ongoing processes.  Historical Logs  Grants access to a comprehensive history of logs for reference and troubleshooting.   Does the Management UI require the Smart Agent to be installed?  No, Smart Agent does not have to be installed. However, the advantage of having the Smart Agent installed on the machine alongside the language agents is to have deeper insights and management capabilities.   When Smart Agent is installed, you will be able to:  Inventory functionality, which in the initial release includes agent status (Out of Date, Update Available, Latest, Unknown), the agent’s version, running tasks, and historical events  Push-button management of upgrades and rollbacks on the machine    Does deploying the Smart Agent require uninstalling existing agents, or is this an incremental process that builds from the existing installed base?  This is an incremental process; you don't need to uninstall anything.   Step 1: Install Smart Agent on the host.   Step 2: Manage your existing and new language agents through the UI.  Will the Smart Agent for agent management have an impact on host compute consumption? If so, what is the expected additional compute load?  While there will be some processing that requires computing with regards to the Smart Agent being installed, it is thought to have no more impact than what an agent does today.    Can users turn off agent management if they choose to do so?  No. Because the Smart Agent is essential for maintaining agent lifecycle workflows, we haven't seen a significant need to develop this particular capability at this time. Can Agent Management also be used to migrate from AppDynamics On-prem to SaaS What if you need to upgrade an agent while also repointing it to communicate with a new SaaS Controller?  Yes, you can use Agent Management to migrate from On-prem to SaaS with Ansible scripts. However, supporting this with a UI is under discussion.    Does agent installation equate with app instrumentation?  No, you need to restart the app for it to be instrumented.   The agent is automatically attached to the application in the cases of Tomcat, WebLogic, Glassfish, and Sprint  In other cases, you need to attach the agent for new installations as they did before Smart Agent by modifying the startup parameters.     What will happen when an existing agent is running as a service?   Existing agents running as a service will be backed up during the new agent version deployment and the same configuration will be used upon update. Nothing changes as to how the service is installed or is run.  Back to TOC     Licenses and Packages  FORUM LINK | Join the discussion on this topic Do I need to buy new licenses for this solution in addition to my existing licenses? There is no additional cost to use Smart Agent.    Is the solution license-aware in that it includes logic to prevent me from over-deploying agents compared to my entitlements?    The current release of Smart Agent does not prevent the over-deployment of agents in relation to your entitlements. However, this functionality is under serious consideration for future updates.  Is Smart Agent officially supported by Cisco AppDynamics if I have problems?   Yes, Smart Agent is officially supported.  How can you manage multiple license keys and utilize the charge-back license model?  To manage multiple license keys for the charge-back license model, you will be able to specify a custom license key in the Additional Configuration section during deployment.  Back to TOC    Supported Environments  FORUM LINK | Join the discussion on this topic What agent flavors are supported?  As of November 2023, the following six agents are supported:  Java  Node.js  Python  PHP  Apache Webserver  Machine Agent   In upcoming releases, look for support for .NET Agent, DBMon and NetViz, among others. Check the documentation for the most current list of supported agents.   What features are supported in simplified agent management?  Supported simplified agent management  Agent installs  Upgrades and rollback via the agent management UI & CLI Management options Bulk management options  Tasks in Progress and History tabs  Logs for troubleshooting Agent Inventory Running inventory of running agents with respective versions  Grid export – with end-to-end accounting of agents running and communicating with the Controller Filtering  New filters include:  Out-dated agents  Available updates by application Auto-attach for Java frameworks *  Auto-attach is supported for Java frameworks (Tomcat, Sprint, WebLogic, Glassfish) and Node.js. With it, admins only need to restart the application because the Smart Agent will know what startup parameters are needed. Otherwise, the startup script must be modified manually.    Are all the Java APM documented deployment scenarios (for different frameworks) supported?  Yes    Is auto-attach supported for Java deployments?  Tomcat, WebLogic, Glassfish, and Spring are supported with this release, and many others are soon to come. Node.js is also supported.    Does Smart Agent support multiple instances of language agents on a single node?  Yes, each language will leverage an instance of a language agent, which will also be managed by the Smart Agent.  Will the Agent Management UI support the ABAP Agent?  As of the v23.11 release, the ABAP Agent is not included as part of the initial UI rollout. It is under consideration for the longer term.     What about the DB Agent?  Discussions to include the DB Agent are underway. We will apprise the Community of its support in a release.    Are PSA and Machine Agent extensions also in the scope of agent management?  PSA and Machine Agent extensions are part of the investigation phase, so there are no timelines to include them yet.    Does Smart Agent support Docker applications (Java, .NET, Node.js, PHP) that are not running on Kubernetes?   The Smart Agent CLI can be used in building Docker containers. It provides a uniform, automated way of installing App agents into containers during build times.    Will I be able to use Smart Agent to manage agents that run on ECS Fargate or Docker containers?   Yes, by integrating with Smart Agent CLI you can deploy agents during build time.    Can we deploy custom extensions to all existing Machine Agents as mass deployment?  Extensions are not supported as of this first release, but are part of future planning.    How will Smart Agent be supported? Regarding Support, we will follow the standard support process for agents, supporting agent releases one year after their release (n+1).  Back to TOC   Requirements  FORUM LINK | Join the discussion on this topic Is a Smart Agent needed on every host? What are the various ways to deploy the Smart Agent?  Yes, one Smart Agent is needed on every host that has applications participating in APM, as it will manage the application agents. Installation methods include: manual installation use the AppDynamics-provided Ansible playbook meant for Smart Agent use existing agent playbooks use Smart Agent CLI for bulk rollouts of the Smart Agent   Are older agents removed when agents are upgraded?  When agents are upgraded, one prior version of the agent is backed up for use in the event of a rollback.      What network configurations and host permissions are required between host and Controller, as well as access to the internet?  Network configurations and host permissions  Install time  Root permission: Most package managers have an implicit root permission requirement due to needing to write to /etc/environment. When a user has a package manager they use to install without root/sudo, then that will be supported.  Runtime  Depending on which agents are being managed, either root or an account with necessary permissions is required for runtime.  Neither Java nor Node.js require root during runtime.   Machine Agent, Python, PHP, and Webserver all require root, with or without the Smart Agent. Network point of view   CSaaS: The port to connect the Smart Agent to is the same as the port other agents connect on. No new port is exposed. 443/80 are the ports for agents as well as Smart Agent now. On-Prem: A different port, -8030, is used for On-prem    What is the minimum language agent version required for Smart Agent to work?   If you are running agents on a supported version of the technology, agent management through Smart Agent will just work.   In the case of technologies that are End of Life (such as Java 6), Agent Management is not supported. Agent compatibility is the same regardless of how they are installed or managed.   This is not a Controller release requirement because we are just managing agents.   Is unidirectional or bi-directional network communication used with Smart Agent?  Smart Agent uses unidirectional Agent-Controller network communications.  Back to TOC   Strategy...  FORUM LINK | Join the discussion on this topic How does Smart Agent address managing hundreds of new and/or existing agents?  Smart Agent is aimed at supporting existing customers with thousands of agents, without their having to uninstall or replace them.    How does Smart Agent work?   With the Smart Agent, you will automatically be shown all existing agents communicating with the existing Controller. So, there is nothing you need to do before gaining immediate value. Agents will appear on the UI and show whether or not the Smart Agent is also a part of that system. If you add a Smart Agent to these machines, this enhancement gives you the additional options to install, upgrade, and rollback.     Can you install and upgrade Smart Agent via Controller UI?  While initial Smart Agent installation is not available via UI—once Smart Agent is installed, you can upgrade it via the Controller UI. Additionally, you can install the Smart Agent through the CLI.    Is there a way to install Smart Agent at the centralized location and manage the agents remotely?   No; Smart Agent should reside on every host that needs to be managed. However, with the CLI, one can manage Smart Agent tasks remotely.    Would Simplified agent management deployments using Smart Agent still require application restarts for the newly deployed agent?  Yes, Simplified agent management simply orchestrates the agent deployments, updates, and rollback operations. Since agents themselves require a restart, the intent for Agent management is to also manage the automation for that aspect but Smart Agent will not restart the applications themselves.    Does Smart Agent have the option of deploying a custom customer agent build—for example on AIX-install that bundles customer JRE, etc.?  Not in this first release. We are working towards testing this in the next release.    What are the differences between the .NET Config Management tool, Smart Agent, and agent management?  Agent management is a set of solutions we offer to customers to simplify agent lifecycle management. Smart Agent is a part of agent management that manages agents on the host.   .NET Config management helps with config file management of .NET agents on machines.    What will happen if the user has FSO Operator and Collector installed on a Cluster and the user is trying to install the Cluster Agent on the same cluster?  The Cluster agent is outside the scope of this release but is being discussed for a later release. Should you need clustering support, the CLI can be used during build time.  Back to TOC   Tooling Pipeline Guidelines  FORUM LINK | Join the discussion on this topic Does this support existing CI/CD pipelines?  Yes, indeed...you can integrate into an existing CI/CD pipeline and are encouraged to do so. Scripting is supported by this effort, and, with the Controller UI release, we are focusing more on supporting larger existing deployments with the capability of adding a Smart Agent to help manage upgrade, rollback, and deployment needs. In a subsequent phase, users will be able to take full advantage of auto-discovery, where all they will do is install Smart Agent, and it will automatically detect which applications need instrumenting then apply the appropriate instrumentation based on those applications’ needs.    How does the Agent Management's UI link to our requirement of using Ansible, Chef, Puppet, etc.?  The scripts serve the CI/CD use case. Though the Agent Management console will use Ansible scripts under the hood, users will not need to engage with scripts.     What prerequisites do users need to adapt? For example, must they have Ansible Tower, a specific PoP, or have used Ansible scripts?  To get started, install a single Smart Agent on each respective host as a one-time activity. After this manual or scripted installation, subsequent maintenance of the Smart Agent can be accomplished from the UI through Upgrades.  There is no Ansible requirement.  Back to TOC   Agent Management User Interface  FORUM LINK | Join the discussion on this topic How do I know whether the Smart Agent is installed?  If the Smart Agent is installed, you can view it on the Controller Agent Management console/UI.    Can we to schedule upgrades?  Though upgrade scheduling is not part of this release, there are plans to provide this functionality in a later release.    What about RBAC?   Role-based access permissions that can be assigned to users will be available in the next release:  ITOPS Admin – can perform all the operations for all the applications.  AppOps Owner – has permission to perform actions to only their application.  DevOps – can view inventory for their applications but can’t perform any operations.    Can data in the Agent Management Controller page be exported?  Yes. You can export the data in the Agent Management Controller with the export grid function.    Will we be able to modify app, tier, and node names for the agents in the Controller?  Yes, you will be able to modify app, tier, and node names in the Controller.     Does the User Interface include information that will help me understand when agents were installed or updated—for instance, a date or timestamp?  Yes.    Is there an option for bulk installing and upgrading Java Agent, Machine Agent, EUM, and others together?  You cannot install different agents at the same time, nor can you perform bulk upgrades of different types at the same time. You can only install or upgrade one type of agent in bulk, like all Java agents.  Can we monitor the uptime of Smart Agent as we do for the app agents? If so, can we create health rules on Smart Agent uptime?  This release doesn’t have Smart Agent self-monitoring. However, we are planning to add this feature in the subsequent phases.    Can we group agents by policy, in such a way as to enable one config change for multiple agents at once?   For example, "AWS Linux 2.0" or "Application Servers"? We are considering whether to tag the hosts in a later release.   In cases where app owners also own the servers, how can clashes between Controller and server admins be avoided?  If the agent is currently being updated and someone tries to update the same agent while the upgrade is in progress, they will receive a warning notification. HTTP_PROXY can be set as an env variable in the Smart Agent, and it would be honored.    How will Smart Agent work for applications like WebSphere, where the JA is passed via User Interface only, or via wrapper only? You can provide values in the Additional Config section.    Do we have an Auditing Capability?  Not at this time. We intend to include auditing capability in an upcoming release.  Is there a rollback-to-the-last-version capability in case a mass rollout causes unintended monitoring issues? Yes, this is a part of the v23.11.0 release.  What tools are available to support troubleshooting failed agent installations and upgrades? Yes, readable deployment and upgrade logs are available in the Ansible if the Controller is using Ansible, or through the UI when using the UI with the Smart Agent.    For SaaS or On-prem cases, can I customize the location from which deployment images are sourced? What security will be placed around selected images and Image repositories? An upcoming release will enable customers to stage deployment from a customized location, which should also support security concerns.  Back to TOC   Can we use the same Smart Agent for Kubernetes deployments?  No, Smart Agent cannot yet be used in Kubernetes deployments.   However, Smart Agent CLI can be used in container deployments with Docker. It provides a uniform automated way of installing App agents into containers during build times.    Can we upgrade two PHP agents running on the same server for different PHP runtime versions with Smart Agent?  Yes.  Back to TOC   Getting Started: Installation FORUM LINK | Join the discussion on this topic Would the customer be able to install the agents manually even if the Smart Agent is installed on those hosts?  Yes. Since they are registered to the Controller upon installing them manually, the agents will then be able to be managed by the Smart Agent as well.     Does the CLI tool use SSH?  Yes, CLI uses SSH or WinRM in the case of remote host Smart Agent installations.     Do you need to deploy an SSH certificate to use Smart Agent?  No certificate exchange is involved with regard to Smart Agent.    Does the Smart Agent require root access or the creation of a user or group? If so, which agents are affected? Does this user have sudo privileges? Yes, it uses root. This is because some of the agents - Python, PHP, Apache, and Machine Agent are currently installed as root. So, for Smart Agent to manage these agents, we need root permissions. If the host only has Java or Node.js Agents, then we need not have root access to run them.   During agent installation or upgrade using Smart Agent, does it also restart the applications? Or will there be an option to restart applications?  No, application restarts are not included.  Back to TOC   Custom configuration files, monitors, and extensions  FORUM LINK | Join the discussion on this topic During brownfield deployment, how are extensions or monitors going to be managed—for example, Analytics Agent as an extension to Machine Agent.  Machine Agent extensions are carried forward to the next version during upgrades. This release does not offer management to extensions on their own. The Analytics Agent is not supported in the v23.11.0 release. For the latest information, see the Supported Agents section in the documentation.    During a brownfield deployment, does Smart Agent retain existing custom config changes on the agent—correlation XML, ISDK deployed?  Yes.  Back to TOC   Additional resources  In the Community Smart Agent lifecycle management reimagined Exploring an APM agent upgrade scenario with Smart Agent  Exploring an APM agent installation scenario with Smart Agent  Smart Agent: How easy is it? — a short, clickable demo NEW! Just one Smart Agent: unlimited agent lifecycle control — overview video (9 minutes) Smart Agent Forum — add your own comments and insights here In the Documentation: Agent Management documentation  Smart Agent documentation  Smart Agent Quick Start Guide  Smart Agent Command Line Utility (CLI) documentation  
Hi, I'm trying to register for the Intro to Splunk SOAR course but when I click on it, it just stays loading. Anyone know how to fix this or who to contact about it?
Recently configured a new input that has successfully ingesting logs but appears to be working intermittently. There is large gaps in logs that we have confirmed are present and being created regular... See more...
Recently configured a new input that has successfully ingesting logs but appears to be working intermittently. There is large gaps in logs that we have confirmed are present and being created regularly from the source server. Example : Logs are captured 8th December and 16 December only. So, here 9th December to 15nth December logs are not captured We have created custom app on our deployment server and push that app across all the deployment slaves. The data flow is coming from source-->Universal forwarder-->Heavy forwarder--> Splunk cloud we have created Inputs.conf  [monitor://F:\Polarion\data\logs\main\*.log.*] sourcetype = catalina index = ito_app disabled = false ignoreOlderThan = 7d initCrcLength = 10000 Please help on the issue Thank you