Hi @ibrahim1 , Site A HF as DS? Short answer: NO - network blocks prevent Site B UF phonehoming to it (8089 outbound needed). UFs poll DS every 30min-ish, same port issue as LM.​ For single UF intermediate: Honestly dont need DS much. Configs super simple: Manual SCP/ansible/rsync changes once a month? Same effort w/ or w/o DS for one box. No biggie. If multiple Site B UFs (or source UFs too): Phonehome workaround: Site B UF → Site A HF:8089 blocked? Use Deployment Server Cluster but needs shared storage between DSs + load balancer. Overkill for forwarders.​ Forwarder mgmt proxy hack (what I've done): Site A HF relays configs? Nah Splunk no built-in for that. But script it: Ansible/Puppet from central → Site A HF Site A HF cron job bundles apps → sends via tcpout to Site B UF (as cooked data) Site B UF inputs.conf catches it, extracts/unpacks to etc/apps/ Kinda janky but works airgapped. Best simple way: Keep source UFs phonehoming to Site A DS/HF (if u open 8089 Site B sources -> Site A). Intermediate UF stays dumb/manual. Or stick w/ HF + 8089 whitelist (one port, easy approval). Single intermediate? Skip DS drama, manual configs fine. Multiple? Ansible > DS in seg nets IMO. ... View more

Hey @ibrahim1 , tough spot with that network seg - seen it before! Your Site B HF def needs Enterprise license cuz its parsing/processing data. UFs are free but HFs gotta phone home to LM. Here's what actually works: Option 1: UF Intermediate Forwarder (cleanest - my rec) Generally most of the time we follow "ntermediate Forwarder" setup to route logs form multiple network sites,  Replace Site B HF with Universal Forwarder. Always free, no LM contact EVER needed. ## Example conf for New Site B UF, change as per your details inputs.conf: [splunktcp://9997] disabled = 0 outputs.conf: [tcpout] defaultGroup = sitea_hf [tcpout:sitea_hf] server = siteA_HF_IP:9997   Source UFs point to new Site B UF:9997. Site A HF does all parsing. Data flows: UF (Site B)→UF(Site B)→HF(Site A)→indexers(Site A). Worked perfect in my DMZ setup. Lose Deployment Server role tho - move that elsewhere or use UF phonehome. Option 2: Selective 8089 firewall rule (if u must keep HF) Ask netsec for just SiteB_HF_IP → SiteA_LM_IP:8089. License heartbeat only (every 30min). Security teams usually approve - low risk vs full mgmt ports. # server.conf at Site B HF [license] master_uri = https://siteA_LM:8089 Option 3: License pooling (if 8089 opens up) Create "forwarders" pool on LM, add Site B HF as slave. Still needs 8089 tho. What I'd do: Go UF intermediate. Zero license drama, lighter resource use, same data flow. Test on one source first  Please give karma 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @jatin3101 , no worries being new to this - SHC captain stuff can be tricky at first. I'll try to break it down simple for your questions what I know so far. Check the official docs too for the full deets. Q1: Election with 4 members left (from 5) Yeah, your guess on the random timer is spot on. All members have a random election timer (like election_timeout_ms, defaults to something around 60s I think). The one that finishes first says "hey vote for me" to everyone - including the dead captain's spot. Needs majority of total members, so for 5 total, thats 3 votes. The 4 alive ones can give those 3 easy, even if captain is down. It worked like that in my 5-node cluster when we lost one. https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.2/configuration-file-reference/9.2.11-configuration-file-reference/server.conf#:~:text=members.%0A*%20Default%3A%20false-,election_timeout_ms,-%3D%20%3Cpositive_integer%3E%0A*%20The%20amount election_timeout_ms = <positive_integer> * The amount of time, in milliseconds, that a member waits before trying to become the captain. * Note that modifying this value can alter the heartbeat period (See election_timeout_2_hb_ratio for further details) * A very low value of election_timeout_ms can lead to unnecessary captain elections. * Default: 60000 (1 minute) ​ Q2: Why odd number recommended over even like 6? Majority is always (N/2)+1 of total members, no matter how many alive. For 6, needs 4 votes. If 3 go down, you're stuck at 3 alive - no captain possible till more recover. With odd like 5 (needs 3), losing 2 leaves 3 alive, still elects fine. Even numbers risk split quorums easier, like 3-3. Docs push 3,5,7 etc for that reason. Not that 6 never works, but riskier.​ Q3: That circular voting thing Nah, not how it works at all. It's not pre-assigned votes like 1->2 etc. Candidate asks all members (alive ones respond), they vote yes/no based on rules (like preferred_captain, in-sync status). First timer proposer usually wins if quorum ok, cuz others agree quick. No cycle voting possible. ​ Hope that clears it up! Heres the main doc link: Splunk Official Doc: https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.2/manage-search-head-clustering/control-captaincy Old article from HurricaneLabs but good one: https://hurricanelabs.com/splunk-tutorials/the-myth-of-the-three-member-search-head-cluster/ Please give karma 👍 for support 😁 happly splunking .... 😎   ... View more

Yes, you can definitely automate Splunk installation without user interaction. The login prompt you are seeing isn't for Splunk.com, it is asking you to create a local admin account for that Splunk instance. Once you've downloaded the MSI file from Splunk.com (one-time login required), you can deploy it silently using below command-line parameters. The basic command looks like this: msiexec.exe /i splunk.msi AGREETOLICENSE=Yes SPLUNKUSERNAME=admin SPLUNKPASSWORD=YourPassword123! /quiet Splunk Doc that you can refer: https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/9.4/install-the-universal-forwarder/install-a-windows-universal-forwarder#install-the-universal-forwarder-silently-agree-to-the-license-and-set-the-forwarder-admin-credentials-to-splunkadminchng3d-0  The key is using AGREETOLICENSE=Yes and setting SPLUNKPASSWORD to skip the interactive setup. You can push this via Group Policy, SCCM, or even a PowerShell script to automate deployments across multiple computers. This way, the installation completes silently in the background, and employees just need to open Splunk and log in with the credentials you've set. You might also want to use INSTALLDIR="C:\Splunk" to avoid path length issues with the default Program Files location. For ongoing management, Splunk's deployment server can handle configuration updates after the initial install. I hope this will fit your requirement. Please give 👍 for support 😁 happly splunking .... 😎   ... View more

I've seen this exact issue before with Splunk Universal Forwarders. The "splunkd.pid doesn't exist" error combined with the "tcp_conn_open_afux ossocket_connect failed" messages typically happens when there's a conflict between how the Splunk process is started and managed. Based on your description, this is likely one of two issues: a. Duplicate systemd service files causing a "split brain" situation b. Permission problems with the Splunk installation directory For the first issue, check if you have duplicate service definitions: ls -la /usr/lib/systemd/system/SplunkForwarder.service ls -la /etc/systemd/system/SplunkForwarder.service If both exist, that's causing your problem! The one in /etc/systemd/system takes precedence, and they might have different user/permission settings. You can fix this by: sudo rm /etc/systemd/system/SplunkForwarder.service sudo systemctl daemon-reload sudo systemctl restart SplunkForwarder If that doesn't work, check the ownership of your Splunk files: ls -la /opt/splunkforwarder Make sure everything is owned by the correct user (typically splunk:splunk). If permissions are wrong, you can fix with: chown -R splunk:splunk /opt/splunkforwarder As a last resort, the complete reinstall approach works well: sudo systemctl stop SplunkForwarder sudo yum remove splunk* sudo rm -rf /opt/splunkforwarder Then reinstall the forwarder and configure it properly. I've had good success with this approach when dealing with these mysterious pid and socket connection errors. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @Na_Kang_Lim, The TrackMe App has a default data retention/tracking period setting that's likely causing you to only see hosts with data_last_time_seen values after 04/03. This is controlled by the "trackme_max_data_tracker_history" macro which defaults to 30 days of history retention. You can modify this setting to include older hosts: a. Navigate to Settings > Advanced Search > Search macros b. Find the macro "trackme_max_data_tracker_history" c. By default, it's set to "-30d" which means it only tracks 30 days of history d. Modify this value to extend the period (e.g., "-60d" or "-90d") e. Save your changes and wait for the next scheduled collection to run Additionally, check these potential causes: a. The TrackMe collection might have been reset or reinstalled around 04/03 b. There might be a tracker purge operation scheduled that removes older entries c. The "Data Sampling" feature might be enabled with a custom time range You can verify current settings with: ``` | rest /servicesNS/-/trackme/configs/conf-trackme/trackme_gen_settings | table title value ``` If you need to force TrackMe to rebuild its tracking data: ``` | outputlookup trackme_host_monitoring ``` Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @MsF-2000,   The issue you're experiencing with Dashboard Studio PDF exports showing empty panels is likely due to timing problems. When exporting dashboards to PDF, Splunk uses Chromium which has default timeouts that may not allow enough time for your searches to complete. Here are solutions to try: 1. Adjust the Chromium timeout settings in limits.conf: A- Increase `render_chromium_timeout` (default is 30 seconds) B- Set `render_chromium_screenshot_delay` to allow more time for searches to complete 2. Optimize your underlying searches: A- Ensure searches use acceleration features when possible B- Consider using report acceleration or summary indexing for complex queries C- Check if any individual panels take longer than 30 seconds to load 3. Try scheduled PDF delivery instead of on-demand export: A- Configure scheduled reports with PDF delivery B- These often provide more time for search completion than interactive exports 4. For Dashboard Studio specifically: A- Avoid using global inputs when possible as they can delay rendering B- Consider converting complex panels to Simple XML if persistent problems occur If modifying limits.conf settings, contact Splunk Cloud Support as they'll need to make these changes for you. Has anyone checked how long the dashboard takes to fully load all data when viewed in the browser? ... View more

Hi @dineshchoudhary, When your "Ignored Messages" configuration isn't working to filter out EventLogger errors in AppDynamics, you need a more comprehensive approach. Here are several methods to stop those noisy events: ## Method 1: Correct Configuration of Error Detection Rules The way you've configured the ignored message might not be matching correctly. Try these variations: 1. Use wildcards in your ignore pattern: a. Use "xxxx.Logs.EventLogger*" instead of the exact message b. Try "*EventLogger*Error*" to match more broadly c. Make sure there are no hidden characters in your pattern 2. Check case sensitivity: a. AppDynamics error matching can be case-sensitive b. Try matching with both uppercase and lowercase variations ## Method 2: Agent Configuration File Approach If the UI-based configuration isn't working, modify the agent config file directly: 1. Locate your agent configuration file: a. For .NET agent: Look for "config.xml" file in the installation directory b. Typically found in "C:\ProgramData\AppDynamics\DotNetAgent\Config" 2. Add the error exclusion in the configuration file: ```xml <configuration> <errorDetection> <ignoredExceptions> <exception> <exceptionType>xxxx.Logs.EventLogger</exceptionType> <exceptionMessage>Error Message: Error</exceptionMessage> </exception> </ignoredExceptions> </errorDetection> </configuration> ``` 3. Restart the application and the AppDynamics agent after making changes ## Method 3: Code-Based Solution If you have access to the application code, consider modifying the logging implementation: 1. In your .NET application, modify the EventLogger class: a. Add logic to prevent reporting these specific errors to AppDynamics b. Use AppDynamics API to exclude certain log messages ```csharp // Example code to prevent AppD from capturing certain errors try { // Your code that might log errors } catch (Exception ex) { // Use AppDynamics API to mark as "do not report" AppDynamics.Agent.Transaction.GetCorrelatingTransaction().DoNotReportAsError(); // Continue with your regular logging EventLogger.LogError("Error Message: Error"); } ``` ## Method 4: Application Tier Settings You might need to adjust settings at the tier level: 1. Navigate to your application configuration: a. Open AppDynamics Controller UI b. Go to Applications > [Your Application] > Configuration > Error Detection 2. Check both "Errors to Ignore" and "Custom Error Detection" sections: a. Add your pattern in both locations b. Use "Contains" match criteria rather than "Equals" 3. Apply changes to the specific tier where your .NET application is running If these approaches don't resolve the issue, the error messages might be coming from a different source than expected or there might be a configuration sync issue between your agent and the controller. Please give 👍 for support & Mark as solution 😁 happly splunking .... 😎 ... View more

Hi @molla  This text wrapping change is indeed a behavior change in Splunk Dashboard Studio in version 9.4.2. It's part of Splunk's ongoing improvements to make dashboards more responsive and readable across different screen sizes, but I understand it may not be what you're looking for in your specific dashboards. To revert to the previous behavior where text is cut off with ellipses rather than wrapped, you have a few options: ## For Text Visualization Elements: 1. Use CSS to override the wrapping behavior: a. Click on the text element in Dashboard Studio b. Go to the Style tab c. Click on "Advanced" at the bottom d. Add the following CSS: ``` .wrapped-text { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; } ``` e. Apply the custom CSS class to your text element 2. Set a fixed width for the text container: a. Select the containing element (panel or section) b. Set a specific width (rather than responsive/percentage-based) c. This will cause text to behave more predictably ## For Table Elements: 1. Modify cell behavior: a. Select your table visualization b. Click on the "Format" tab c. Find the specific column you want to modify d. Click "Cell" settings e. Under "Text Overflow", select "Clip" or "Ellipsis" instead of "Wrap" 2. Apply custom CSS to table cells: a. Select the table b. Go to the Style tab c. Click on "Advanced" d. Add the following CSS: ``` td.splunk-table-cell { white-space: nowrap !important; overflow: hidden !important; text-overflow: ellipsis !important; } ``` ## For Single Value Visualizations: 1. Adjust truncation settings: a. Select the single value visualization b. Go to the Format tab c. Find the "Truncate" option d. Set it to "End" and specify the desired character limit If these methods don't work for your specific dashboard elements, you may need to provide more details about which specific visualization types are showing the unwanted wrapping behavior. Note that this is a UI/rendering change in 9.4.2, so there's no global setting to revert all text handling to the previous version's behavior. Each element needs to be adjusted individually. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @asadnafees138  When backing up Splunk logs to AWS S3, the format depends on which method you use for the backup. Here are the available methods and their corresponding formats: ## Ingest Actions If you're using Ingest Actions to send data to S3, you have three format options: 1. Raw format a. This preserves the original format of your data exactly as it was ingested b. Best for maintaining complete fidelity with source data c. May require additional parsing when retrieving 2. Newline-delimited JSON (ndjson) a. Each event is a separate JSON object on a new line b. Includes both the raw event and extracted fields c. Best option if you plan to use Federated Search with S3 later d. Easily parseable by many analytics tools 3. JSON format a. Standard JSON format with events in an array structure b. Includes metadata and extracted fields c. Good for interoperability with other systems ## Edge Processor and Ingest Processor If using Edge Processor or Ingest Processor: 1. JSON format a. Default format b. Structured in HTTP Event Collector (HEC) compatible format c. Includes event data and metadata 2. Parquet format (Ingest Processor only) a. Columnar storage format b. Offers better compression and query performance c. Excellent for analytical workloads d. Supported by many big data tools ## Frozen Data Archive If archiving frozen buckets to S3: 1. Splunk proprietary format a. Data stored in Splunk's internal format (tsidx and raw files) b. Requires Splunk to read and interpret c. Best for data you might want to thaw back into Splunk later 2. Custom format (with cold2frozen scripts) a. You can customize how data is exported using scripts b. Can transform to various formats including CSV, JSON, etc. ## Recommendations Based on your needs for future retrieval and readability: 1. If you need the data to be easily readable by other systems: a. Use Ingest Actions with ndjson format b. Or Ingest Processor with Parquet format for analytical workloads 2. If you might want to re-ingest into Splunk: a. ndjson format is easiest for re-ingestion b. Frozen bucket archives can be thawed but only within Splunk 3. If storage efficiency is a priority: a. Parquet format (via Ingest Processor) offers the best compression b. ndjson is a good balance between readability and size For comprehensive documentation, refer to: Ingest Actions: https://docs.splunk.com/Documentation/SplunkCloud/latest/IngestActions/S3Destination Edge Processor: https://docs.splunk.com/Documentation/SplunkCloud/latest/EdgeProcessor/AmazonS3Destination Ingest Processor: https://docs.splunk.com/Documentation/SplunkCloud/latest/IngestProcessor/AmazonS3Destin Data Self-Storage: https://docs.splunk.com/Documentation/SplunkCloud/latest/Admin/DataSelfStorage Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @Uday, There are several approaches to create a server status dashboard in Splunk when you don't have explicit "server up/down" logs. Here are the most effective methods: ## Method 1: Check for Recent Log Activity This is the simplest approach - if a server is sending logs, it's probably up: ``` | metadata type=hosts index=* | search host=* | eval lastTime=strftime(recentTime,"%Y-%m-%d %H:%M:%S") | eval status=if(now()-recentTime < 600, "UP", "DOWN") | table host lastTime status | sort host ``` Customize the time threshold (600 seconds = 10 minutes) based on your expected log frequency. ## Method 2: Using Rangemap for Visualization Use rangemap to assign colors to status values: ``` | metadata type=hosts index=* | search host=* | eval lastTime=strftime(recentTime,"%Y-%m-%d %H:%M:%S") | eval seconds_since_last_log=now()-recentTime | eval status=if(seconds_since_last_log < 600, "UP", "DOWN") | rangemap field=status up="0-0" down="1-1" | table host lastTime status range | sort host ``` For dashboard visualization, you'll need to add: 1. A CSS file (table_decorations.css) with content: ```css .severe { background-color: #dc4e41 !important; color: white !important; } .low { background-color: #65a637 !important; color: white !important; } ``` 2. A JavaScript file (table_icons_rangemap.js) with content: ```javascript require([ 'underscore', 'jquery', 'splunkjs/mvc', 'splunkjs/mvc/tableview', 'splunkjs/mvc/simplexml/ready!' ], function(_, $, mvc, TableView) { var CustomRangeRenderer = TableView.BaseCellRenderer.extend({ canRender: function(cell) { return cell.field === 'range'; }, render: function($td, cell) { var value = cell.value; if(value === "severe") { $td.addClass('severe'); $td.html('Down'); } else if(value === "low") { $td.addClass('low'); $td.html('Up'); } return $td; } }); mvc.Components.get('table1').getVisualization(function(tableView) { tableView.addCellRenderer(new CustomRangeRenderer()); tableView.render(); }); }); ``` 3. Dashboard XML that includes these files: ```xml <form script="table_icons_rangemap.js" stylesheet="table_decorations.css"> <label>Server Status Dashboard</label> <fieldset submitButton="false"> <input type="time" token="field1"> <label></label> <default> <earliest>-60m@m</earliest> <latest>now</latest> </default> </input> </fieldset> <row> <panel> <table id="table1"> <search> <query>| metadata type=hosts index=* | search host=* | eval lastTime=strftime(recentTime,"%Y-%m-%d %H:%M:%S") | eval seconds_since_last_log=now()-recentTime | eval status=if(seconds_since_last_log < 600, "UP", "DOWN") | rangemap field=status up="0-0" down="1-1" | table host lastTime status range | sort host</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> </search> <option name="drilldown">none</option> </table> </panel> </row> </form> ``` ## Method 3: Include All Expected Servers To also show servers that aren't sending logs at all, use a lookup with all expected servers: ``` | inputlookup your_servers.csv | append [| metadata type=hosts index=*] | stats max(recentTime) as recentTime by host | eval lastTime=if(isnotnull(recentTime),strftime(recentTime,"%Y-%m-%d %H:%M:%S"),"Never") | eval seconds_since_last_log=if(isnotnull(recentTime),now()-recentTime,999999) | eval status=if(seconds_since_last_log < 600, "UP", "DOWN") | rangemap field=status up="0-0" down="1-1" | table host lastTime status range | sort host ``` ## Method 4: Advanced Server Status Check (Recommended for Critical Systems) If exact server status is critical, create a scheduled search that sends heartbeats from each server and alerts when they're missing: 1. Create a small script on each server that sends a heartbeat every few minutes: ``` index=server_status sourcetype=heartbeat host=$HOSTNAME$ status=ALIVE ``` 2. Then use this search for your dashboard: ``` | inputlookup your_servers.csv | map search="search earliest=-10m latest=now index=server_status sourcetype=heartbeat host=$host$ | head 1 | fields host" | fillnull value="DOWN" status | eval status=if(host=="NULL","DOWN","UP") | rangemap field=status up="0-0" down="1-1" | table host status range ``` This solution is more accurate than just checking for any logs, as it specifically monitors for heartbeat messages. Remember to place your CSS and JS files in the /appserver/static/ directory of your app, and restart Splunk after adding them. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @genesiusj, Based on your description, you're dealing with a time series forecasting problem where you want to predict future user access patterns on Sundays. For this type of scenario in MLTK, I would recommend the following algorithms: ## Recommended Algorithms 1. Prophet a. Excellent for time series data with strong seasonal patterns (like your Sunday-only data) b. Handles missing values well, which is useful since many users may have zero counts on certain days c. Can capture multiple seasonal patterns (weekly, monthly, yearly) d. Works well when you have 6 months of historical data 2. ARIMA (AutoRegressive Integrated Moving Average) a. Good for detecting patterns and generating forecasts based on historical values b. Works well for data that shows trends over time c. Can handle seasonal patterns with the seasonal variant (SARIMA) d. Requires stationary data (you might need to difference your time series) ## Implementation Approach For your specific use case with 1000 users, I would recommend using a separate model for each user who has sufficient historical data. Here's how you could implement this with Prophet: ``` | inputlookup your_lookup.csv | where DATE >= "2020-01-05" AND DATE <= "2020-06-28" | rename DATE as ds, COUNT as y | fit Prophet future_timespan=26 from ds y by USER | where isnull(y) | eval date_str=strftime(ds, "%Y-%m-%d") | rename ds as DATE | fields DATE USER yhat yhat_lower yhat_upper | eval predicted_count = round(yhat) | fields DATE USER predicted_count ``` For comparison with actual values: ``` | inputlookup your_lookup.csv | where DATE >= "2020-07-05" AND DATE <= "2020-12-27" | join type=left USER DATE [| inputlookup your_lookup.csv | where DATE >= "2020-01-05" AND DATE <= "2020-06-28" | rename DATE as ds, COUNT as y | fit Prophet future_timespan=26 from ds y by USER | where isnull(y) | eval DATE=strftime(ds, "%Y-%m-%d") | fields DATE USER yhat | eval predicted_count = round(yhat)] | rename COUNT as actual_count | eval error = abs(actual_count - predicted_count) | eval error_percentage = if(actual_count=0, if(predicted_count=0, 0, 100), round((error/actual_count)*100, 2)) ``` ## Handling Your Data Structure Since you have 1000 users and 52 Sundays, I have a few recommendations for improving your forecasting: 1. Focus first on users with non-zero access patterns a. Many users might have sparse or no access attempts, which can result in poor models b. Consider filtering to users who accessed the system at least N times during the training period 2. Consider feature engineering a. Add month and quarter features to help the model capture broader seasonal patterns b. Include special event indicators if certain Sundays might have unusual patterns (holidays, etc.) c. You might want to include a lag feature (access count from previous Sunday) 3. Model evaluation a. Compare MAPE (Mean Absolute Percentage Error) across different users and algorithms b. For users with sparse access patterns, consider MAE (Mean Absolute Error) instead c. Establish a baseline model (like average access count per Sunday) to compare against 4. Alternative approach for sparse data a. For users with very sparse access patterns, consider binary classification b. Predict whether a user will attempt access (yes/no) rather than count c. Use algorithms like Logistic Regression or Random Forest for this approach Hope this helps point you in the right direction! With 6 months of training data focused on weekly patterns, Prophet is likely your best starting point. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @token2, To answer your questions about the VMware add-ons: ## Do You Need Both Add-ons? No, you don't necessarily need both add-ons in an environment with vCenter, but using both provides more complete visibility. Here's why: 1. Splunk Add-on for vCenter: a. Collects vCenter-specific logs and metrics b. Gathers performance data through the vCenter API c. Collects vCenter Server events, tasks, and alarms d. Can collect some forwarded ESXi logs that vCenter has received (if configured to do so) 2. Splunk Add-on for VMware ESXi: a. Collects ESXi host-specific logs directly from each host b. Captures detailed host-level events that may not all be forwarded to vCenter c. Provides more granular host-level monitoring d. Essential for troubleshooting host-specific issues While vCenter does collect many ESXi logs, it doesn't necessarily collect everything. Some detailed ESXi logs remain local to the hosts and aren't forwarded to vCenter, especially debug-level logs and certain system events. Collecting directly from ESXi hosts gives you more complete visibility. ## Collection Methods for vCenter Add-on Yes, the Splunk Add-on for vCenter typically utilizes both collection methods: 1. Syslog collection: a. For operational logs and events from vCenter b. Requires configuring vCenter to forward logs via syslog 2. API access: a. For performance metrics, inventory, and task/event data b. Requires a vCenter user account with appropriate permissions c. Uses REST API calls to gather data This dual-collection approach gives you both operational logs and rich performance/configuration data. ## Recommended Setup For a complete VMware monitoring solution with vCenter: 1. If complete visibility is important: a. Install both add-ons b. Configure syslog from both vCenter and all ESXi hosts c. Set up API collection from vCenter 2. If you have resource constraints or simpler needs: a. Install the vCenter add-on only b. Ensure vCenter is configured to collect as many ESXi logs as possible c. You'll miss some host-specific details but will have good overall visibility 3. If you have a very large environment: a. Install both add-ons b. Consider selective monitoring of critical ESXi hosts only c. Use the vCenter add-on for broad monitoring and the ESXi add-on for deep dive into important hosts The biggest advantage of using both add-ons is the additional context and detail you get from direct ESXi host monitoring, especially valuable for troubleshooting host-specific issues that might not be fully visible through vCenter alone. Hope this helps clarify your VMware monitoring options in Splunk! Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @asah, No, the traditional Splunk Deployment Server cannot be used to manage Splunk OpenTelemetry (OTel) Collectors running in Kubernetes clusters. Here's why and what alternatives you should consider: ## Why Deployment Server Won't Work 1. **Different Architecture**: Splunk Deployment Server is designed to manage Splunk-specific components like Universal Forwarders and Heavy Forwarders, which use Splunk's proprietary configuration system. The OpenTelemetry Collector uses a completely different configuration approach. 2. **Kubernetes-Native Components**: OTel Collectors running in Kubernetes are typically deployed as Kubernetes resources (Deployments, DaemonSets, etc.) and follow Kubernetes configuration patterns using ConfigMaps or Secrets. 3. **Configuration Format**: OTel Collectors use YAML configurations with a specific schema that's different from Splunk's .conf files. ## Recommended Approaches for Managing OTel Collectors in Kubernetes ### 1. GitOps Workflow (Recommended) Use a GitOps approach with tools like: - Flux or ArgoCD for configuration management - Store your OTel configurations in a Git repository - Use Kubernetes ConfigMaps to mount configurations into your collectors ### 2. Helm Charts The Splunk OpenTelemetry Collector Helm chart provides a manageable way to deploy and configure collectors: ```bash helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart helm install my-splunk-otel splunk-otel-collector-chart/splunk-otel-collector \ --set gateway.enabled=true \ --set clusterName=my-cluster ``` You can create custom values.yaml files for different environments and manage them in your version control system. ### 3. Kubernetes Operator For more sophisticated management, consider using an operator pattern. While there isn't an official OTel operator from Splunk yet, you could implement your own custom operator or use community-developed options. ### 4. Configuration Management Tools Use standard configuration management tools like: - Ansible - Terraform - Puppet/Chef These can apply configuration changes across your Kubernetes clusters in a controlled manner. ## Practical Example Here's a simplified workflow for managing OTel configurations in Kubernetes: 1. Store your base collector config in a Git repo: ```yaml # otel-collector-config.yaml receivers: filelog: include: [/var/log/containers/*.log] processors: batch: timeout: 1s exporters: splunk_hec: token: "${SPLUNK_HEC_TOKEN}" endpoint: "https://your-splunk-cloud-instance.splunkcloud.com:8088" service: pipelines: logs: receivers: [filelog] processors: [batch] exporters: [splunk_hec] ``` 2. Create a ConfigMap in Kubernetes: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config namespace: splunk data: collector.yaml: | receivers: filelog: include: [/var/log/containers/*.log] # Rest of config... ``` 3. Mount the ConfigMap in your OTel Collector deployment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector spec: template: spec: containers: - name: otel-collector image: otel/opentelemetry-collector-contrib:latest volumeMounts: - name: config mountPath: /etc/otel/config.yaml subPath: collector.yaml volumes: - name: config configMap: name: otel-collector-config ``` This approach lets you manage configurations in a Kubernetes-native way, with proper version control and rollout strategies. For more information, I recommend checking the official documentation: - [Splunk OpenTelemetry Collector for Kubernetes](https://github.com/signalfx/splunk-otel-collector-chart) - [OpenTelemetry Collector Configuration](https://opentelemetry.io/docs/collector/configuration/) Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @Cbr1sg ,   Based on the thread and my experience with this issue, the Microsoft Teams Add-on for Splunk has a known deficiency in handling 404 errors properly. Here's what's happening: 1. When the add-on encounters call IDs that no longer exist in Microsoft Teams (returning 404 errors), it fails to remove these IDs from the webhook directory. 2. This causes a build-up of unprocessable call IDs, leading to: a. Continuous error messages in the logs b. Eventually a "401 Unauthorized" error when too many files accumulate (~60K files) c. The add-on completely stops working until restarted The most reliable fix I've found is the following procedure that needs to be performed periodically (some users report every few days): ## Solution Steps: 1. Disable all inputs in this order: a. Call Record input b. User Report input (if configured) c. Subscription input d. Webhook input 2. Clean the KVStore to reset the checkpointer: splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer Note: You need to run this command on the machine where the add-on is installed (usually a heavy forwarder). 3. Re-enable the inputs in this specific order: a. Webhook input b. Subscription input (this will recreate the subscription) c. Call Record input d. User Report input (if used) 4. Additional steps for persistent solution: a. If you're comfortable with scripting, you can create a scheduled task (cron job) to run these steps nightly b. For a more advanced solution, you could create an alert that triggers when "404 Not Found" errors appear in logs ## Scripted Solution Example: Here's a bash script that you could schedule to run nightly: bash #!/bin/bash # Path to Splunk binary SPLUNK_BIN="/opt/splunk/bin/splunk" # Disable inputs $SPLUNK_BIN disable input TA_MS_Teams://call_record $SPLUNK_BIN disable input TA_MS_Teams://user_report $SPLUNK_BIN disable input TA_MS_Teams://subscription $SPLUNK_BIN disable input TA_MS_Teams://webhook # Wait for processes to stop sleep 10 # Clean KVStore $SPLUNK_BIN clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer # Re-enable inputs in correct order $SPLUNK_BIN enable input TA_MS_Teams://webhook sleep 5 $SPLUNK_BIN enable input TA_MS_Teams://subscription sleep 10 $SPLUNK_BIN enable input TA_MS_Teams://call_record $SPLUNK_BIN enable input TA_MS_Teams://user_report echo "Microsoft Teams Add-on inputs reset completed at $(date)" Note that while this is a functional workaround, the root issue is in the add-on's code not properly handling 404 errors. As mentioned by others in the thread, the add-on should ideally be updated to remove call IDs from the webhook directory when they return 404 errors. If you're experiencing this issue frequently, I recommend also opening a support case with Splunk to encourage the development team to address this in a future update of the add-on. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @pramod,   I've worked with the Proofpoint ET Splunk TA in Splunk Cloud, and there's a specific way to handle the authentication for this app. For configuring the Proofpoint ET Intelligence in Splunk Cloud, you need to understand that there's a difference between the "authorization code" and the "Oink code": 1. The API key is what you get from your ET Intelligence subscription. 2. The "authorization code" field in the TA configuration actually requires your ET Intelligence subscription key (sometimes also called "download key"), NOT the Oink code. This is a common confusion point. 3. If you're seeing "None" for the authorization code, it's likely because that field hasn't been properly populated in your account settings on the Proofpoint ET Intelligence portal. Here's how to properly configure it: 1. Log in to your ET Intelligence account at https://threatintel.proofpoint.com/ 2. Navigate to "Account Settings" (usually in the top-right profile menu) 3. Make sure both your API key and subscription key (download key) are available - if your subscription key shows "None", contact Proofpoint support to have it properly provisioned 4. In Splunk Cloud: a. Install the Proofpoint ET Splunk TA through the Splunk Cloud self-service app installation b. Open the app configuration c. Enter your API key in the "API Key" field d Enter your subscription key (download key) in the "Authorization Code" field (NOT the Oink code) e. Save the configuration 5. If you're still getting errors, check the following: a. Look at the _internal index for any API connection errors b. Verify your Splunk Cloud instance has proper outbound connectivity to the Proofpoint ET Intelligence API endpoints c. Confirm with Proofpoint that your subscription is active and properly configured If you're still having issues after trying these steps, you may need to: 1. Submit a support ticket with Proofpoint to verify your account credentials 2. Work with Splunk Cloud support to ensure the app is properly installed and has the right permissions Please give 👍 for support 😁 happly splunking .... 😎 ... View more

Hi @clumicao, Currently, Mission Control does not have the same capability as Enterprise Security to create and share saved filters across users. In Mission Control, filters are saved locally to your browser and user profile, making them user-specific rather than shareable across a team. There are a few workarounds you can use: 1. Document your most useful filters in a shared team document so others can manually recreate them 2. Use Mission Control's Export/Import feature to share filter configurations: a. After creating a filter, click on the "Export" option in the filter panel b. This will download a JSON configuration file c. Share this file with team members d. Other users can import this file using the "Import" option in their filter panel Note that even with the import method, each user would need to import the filter individually, and any updates to the filter would require re-sharing and re-importing. This has been a requested feature, and I recommend submitting it through the Splunk Ideas portal if it would be valuable for your team. The Splunk Mission Control team is regularly enhancing the product based on user feedback. Please give 👍 for support 😁 happly splunking .... 😎 ... View more

For optimizing the AppDynamics Events Service Cluster startup process with self-healing capabilities, I recommend implementing systemd service units with proper configurations. This approach handles all your scenarios: graceful shutdown, unplanned downtime, accidental process termination, and OOM situations. Here's a comprehensive solution: 1. Create systemd service units for both Events Service and Elasticsearch: For Events Service (events-service.service): ``` [Unit] Description=AppDynamics Events Service After=network.target elasticsearch.service Requires=elasticsearch.service [Service] Type=forking User=appdynamics Group=appdynamics WorkingDirectory=/opt/appdynamics/events-service ExecStart=/opt/appdynamics/events-service/bin/events-service.sh start ExecStop=/opt/appdynamics/events-service/bin/events-service.sh stop PIDFile=/opt/appdynamics/events-service/pid.txt TimeoutStartSec=300 TimeoutStopSec=120 # Restart settings for self-healing Restart=always RestartSec=60 # OOM handling OOMScoreAdjust=-900 # Health check script ExecStartPost=/opt/appdynamics/scripts/events-service-health-check.sh [Install] WantedBy=multi-user.target ``` For Elasticsearch (elasticsearch.service): ``` [Unit] Description=Elasticsearch for AppDynamics After=network.target [Service] Type=forking User=appdynamics Group=appdynamics WorkingDirectory=/opt/appdynamics/events-service/elasticsearch ExecStart=/opt/appdynamics/events-service/elasticsearch/bin/elasticsearch -d -p pid ExecStop=/bin/kill -SIGTERM $MAINPID PIDFile=/opt/appdynamics/events-service/elasticsearch/pid # Restart settings for self-healing Restart=always RestartSec=60 # Resource limits for OOM prevention LimitNOFILE=65536 LimitNPROC=4096 LimitMEMLOCK=infinity LimitAS=infinity # OOM handling OOMScoreAdjust=-800 [Install] WantedBy=multi-user.target ``` 2. Create a health check script (events-service-health-check.sh): ```bash #!/bin/bash # Health check for Events Service EVENT_SERVICE_PORT=9080 MAX_RETRIES=3 RETRY_INTERVAL=20 check_events_service() { for i in $(seq 1 $MAX_RETRIES); do if curl -s http://localhost:$EVENT_SERVICE_PORT/healthcheck > /dev/null; then echo "Events Service is running properly." return 0 else echo "Attempt $i: Events Service health check failed. Waiting $RETRY_INTERVAL seconds..." sleep $RETRY_INTERVAL fi done echo "Events Service failed health check after $MAX_RETRIES attempts." return 1 } check_elasticsearch() { for i in $(seq 1 $MAX_RETRIES); do if curl -s http://localhost:9200/_cluster/health | grep -q '"status":"green"\|"status":"yellow"'; then echo "Elasticsearch is running properly." return 0 else echo "Attempt $i: Elasticsearch health check failed. Waiting $RETRY_INTERVAL seconds..." sleep $RETRY_INTERVAL fi done echo "Elasticsearch failed health check after $MAX_RETRIES attempts." return 1 } main() { # Wait for initial startup sleep 30 if ! check_elasticsearch; then echo "Restarting Elasticsearch due to failed health check..." systemctl restart elasticsearch.service fi if ! check_events_service; then echo "Restarting Events Service due to failed health check..." systemctl restart events-service.service fi } main ``` 3. Set up a watchdog script for OOM monitoring (run as a cron job every 5 minutes): ```bash #!/bin/bash LOG_FILE="/var/log/appdynamics/oom-watchdog.log" ES_HEAP_THRESHOLD=90 ES_SERVICE="elasticsearch.service" EVENTS_HEAP_THRESHOLD=90 EVENTS_SERVICE="events-service.service" log_message() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> $LOG_FILE } check_elasticsearch_memory() { ES_MEMORY_PCT=$(ps -C java -o cmd= | grep elasticsearch | grep -o "Xmx[0-9]*[mMgG]" | head -1) ES_CURRENT_HEAP=$(jstat -gc $(pgrep -f elasticsearch) 2>&1 | tail -n 1 | awk '{print ($3+$4+$5+$6+$7+$8)/1024}') if [[ $ES_MEMORY_PCT == *"g"* || $ES_MEMORY_PCT == *"G"* ]]; then ES_MAX_HEAP=${ES_MEMORY_PCT//[^0-9]/} ES_MAX_HEAP=$((ES_MAX_HEAP * 1024)) else ES_MAX_HEAP=${ES_MEMORY_PCT//[^0-9]/} fi ES_HEAP_PCT=$((ES_CURRENT_HEAP * 100 / ES_MAX_HEAP)) if [ $ES_HEAP_PCT -gt $ES_HEAP_THRESHOLD ]; then log_message "Elasticsearch heap usage at ${ES_HEAP_PCT}% - exceeds threshold of ${ES_HEAP_THRESHOLD}%. Restarting service." systemctl restart $ES_SERVICE return 0 fi return 1 } check_events_service_memory() { EVENTS_MEMORY_PCT=$(ps -C java -o cmd= | grep events-service | grep -o "Xmx[0-9]*[mMgG]" | head -1) EVENTS_CURRENT_HEAP=$(jstat -gc $(pgrep -f "events-service" | grep -v grep | head -1) 2>&1 | tail -n 1 | awk '{print ($3+$4+$5+$6+$7+$8)/1024}') if [[ $EVENTS_MEMORY_PCT == *"g"* || $EVENTS_MEMORY_PCT == *"G"* ]]; then EVENTS_MAX_HEAP=${EVENTS_MEMORY_PCT//[^0-9]/} EVENTS_MAX_HEAP=$((EVENTS_MAX_HEAP * 1024)) else EVENTS_MAX_HEAP=${EVENTS_MEMORY_PCT//[^0-9]/} fi EVENTS_HEAP_PCT=$((EVENTS_CURRENT_HEAP * 100 / EVENTS_MAX_HEAP)) if [ $EVENTS_HEAP_PCT -gt $EVENTS_HEAP_THRESHOLD ]; then log_message "Events Service heap usage at ${EVENTS_HEAP_PCT}% - exceeds threshold of ${EVENTS_HEAP_THRESHOLD}%. Restarting service." systemctl restart $EVENTS_SERVICE return 0 fi return 1 } # Check if services are running if ! systemctl is-active --quiet $ES_SERVICE; then log_message "Elasticsearch service is not running. Attempting to start." systemctl start $ES_SERVICE fi if ! systemctl is-active --quiet $EVENTS_SERVICE; then log_message "Events Service is not running. Attempting to start." systemctl start $EVENTS_SERVICE fi # Check memory usage check_elasticsearch_memory check_events_service_memory ``` 4. Enable and start the services: ```bash # Make scripts executable chmod +x /opt/appdynamics/scripts/events-service-health-check.sh chmod +x /opt/appdynamics/scripts/oom-watchdog.sh # Place service files cp elasticsearch.service /etc/systemd/system/ cp events-service.service /etc/systemd/system/ # Reload systemd, enable and start services systemctl daemon-reload systemctl enable elasticsearch.service systemctl enable events-service.service systemctl start elasticsearch.service systemctl start events-service.service # Add the OOM watchdog to cron (crontab -l 2>/dev/null; echo "*/5 * * * * /opt/appdynamics/scripts/oom-watchdog.sh") | crontab - ``` This setup addresses all your scenarios: 1. Graceful Shutdown: The systemd ExecStop commands ensure proper shutdown sequences 2. Unplanned Downtime: The Restart=always ensures services restart after VM reboots 3. Accidental Process Kill: Again, Restart=always handles this automatically 4. OOM Situations: Combination of OOMScoreAdjust, resource limits, and the custom watchdog script You may need to adjust paths and user/group settings to match your environment. This implementation provides comprehensive self-healing for AppDynamics Events Service and Elasticsearch. ... View more

Hi @vishutanuku, For AppDynamics Java agent with Tomcat, you have multiple options to configure proxy settings: OPTION 1: JVM ARGUMENTS ----------------------- You can add the proxy settings as JVM arguments in your Tomcat startup script (catalina.sh/catalina.bat or setenv.sh/setenv.bat): JAVA_OPTS="$JAVA_OPTS -Dappdynamics.http.proxyHost=your_proxy_host -Dappdynamics.http.proxyPort=your_proxy_port" If your proxy requires authentication, also add: JAVA_OPTS="$JAVA_OPTS -Dappdynamics.http.proxyUser=your_proxy_username -Dappdynamics.http.proxyPasswordFile=/path/to/password/file" OPTION 2: CONTROLLER-INFO.XML ---------------------------- Yes, you can add proxy settings directly in the controller-info.xml file, which is located in the agent's conf directory: xml <controller-info> <!-- Other controller settings --> <http-proxy-host>your_proxy_host</http-proxy-host> <http-proxy-port>your_proxy_port</http-proxy-port> <http-proxy-username>your_proxy_username</http-proxy-username> <http-proxy-password-file>/path/to/password/file</http-proxy-password-file> </controller-info> OPTION 3: SYSTEM PROPERTIES -------------------------- You can also set system-wide proxy settings using standard Java system properties:JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=your_proxy_host -Dhttp.proxyPort=your_proxy_port -Dhttp.nonProxyHosts=localhost|127.0.0.1" BEST PRACTICES ------------- 1. For security, store proxy passwords in a separate file and use the password file reference 2. Make sure the agent has read permissions for the password file 3. Restart Tomcat after making these changes 4. Check agent logs for any proxy-related connection issues 5. If using HTTPS for the controller, you may also need HTTPS proxy settings To verify if your proxy settings are working, check the agent logs for successful connection messages to the controller after restart. These configurations work for both AppDynamics Java agent versions 4.x and 21.x+. Please give 👍 for support 😁 happly splunking .... 😎 ... View more