All Posts

Top

All Posts

@splunkreal  The configuration file props.conf will be stored in $SPLUNK_HOME/etc/apps(Heavy forwarder) if you generate it on the deployment server at $SPLUNK_HOME/etc/deployment-apps and send it to... See more...
@splunkreal  The configuration file props.conf will be stored in $SPLUNK_HOME/etc/apps(Heavy forwarder) if you generate it on the deployment server at $SPLUNK_HOME/etc/deployment-apps and send it to the heavy forwarders via the deployment server. Please remove the file before pushing it from the deployment server; otherwise, it will not automatically appear in the /etc/apps (heavy forwarder) when the deployment server is reloaded. It will surpass. https://docs.splunk.com/Documentation/Splunk/9.2.1/Updating/Createdeploymentapps  How to edit a configuration file - Splunk Documentation 
Hi @Muhammad Husnain.Ashfaq, Thank you for coming back to the community and sharing the solution! 
Hi @WILLIAM.GREENE, Have you had a chance to review the reply above? Can you confirm if this has helped you? If not, please jump back into the conversation to keep it going. 
Hello, if we have on DS "app/local" with conf files, is that possible restarting it that it pushes DS "app/local" to HF "app/local" and deletes custom local conf files on HF (created from HF GUI)? ... See more...
Hello, if we have on DS "app/local" with conf files, is that possible restarting it that it pushes DS "app/local" to HF "app/local" and deletes custom local conf files on HF (created from HF GUI)? Thanks.
Yes. Starts with $7. Thanks for the reply
Hi, I was looking for an answer to the same problem, and I came across this older post which kind of confirmed my understanding of the issue and the available solutions: https://community.splunk.co... See more...
Hi, I was looking for an answer to the same problem, and I came across this older post which kind of confirmed my understanding of the issue and the available solutions: https://community.splunk.com/t5/Deployment-Architecture/Why-is-cluster-master-reporting-quot-Cannot-fix-search-count-as/m-p/150441/highlight/true#M5597 Short summary: hot buckets are streamed from the originating indexer to the other indexers in the cluster, but sometimes they get out of sync for various reasons and the CM starts displaying this type of errors. Two ways to fix it: either roll the buckets (via the GUI on the CM, the API endpoint or by performing a rolling restart of the peers) or wait that they naturally roll. In my case, I'll now be investigating why and how we have these "de-synchronisations". On a different note, and perhaps not completely relevant, you indicated your hot buckets to have a retention of 91 days. This seems pretty long to me (not double checked the doc on that but still). There is also the warm stage between hot and cold, I would typically have a shorter period for the hot buckets, and keep them warm for a sensible period, before rolling them cold.
Thank you very much for your help. The code that works  ------- index=firewall event_type="error" [search index=firewall sourcetype="metadata" enforcement_mode=block | dedup host1 | tabl... See more...
Thank you very much for your help. The code that works  ------- index=firewall event_type="error" [search index=firewall sourcetype="metadata" enforcement_mode=block | dedup host1 | table host1 | format] | dedup host | table event_type, host, ip -----------
 Thank You very much @VatsalJagani  I had to click on the version number (link), and there did open what I needed best regards Altin
Hello, How to solve " Events might not be returned in sub-second order due to search memory limits" without increasing the value of the following limits.conf setting:[search]:max_rawsize_perchunk?... See more...
Hello, How to solve " Events might not be returned in sub-second order due to search memory limits" without increasing the value of the following limits.conf setting:[search]:max_rawsize_perchunk? I got a message after I scheduled a query to move more than 150k rows into a summary index. I appreciate your help. Thank you
Hi I'm afraid that in this case there is SHC which is managed by deployer as it should. BUT then someone has installed one app into one member locally from cli or just unpack that file into correct ... See more...
Hi I'm afraid that in this case there is SHC which is managed by deployer as it should. BUT then someone has installed one app into one member locally from cli or just unpack that file into correct app folder? @aasserhifni is this assumption correct? If it it then you are in deep s...t. I have one found this kind of situation and only way how I get rid of it was just disable that app locally. It's not help even install it first by SHC Deployer and then remove it by SHC Deployer. It just sit there. I haven't had time to figure out is there any way to get rid of it from cli or other method. It seems that there is some unknown (at least for me) mechanism how SHC manage this kind of situations. Probably something with kvstore and something on filesystems and something on captain. Maybe you could try to stop whole SHC and then remove that app on member and check if it's still there after start all nodes or not. I cannot test that as that environment was quite busy production with main alerts etc. If that is not helping you, you should ask help from Splunk Support, if they have some way to figure it out? r. Ismo
I have it this way (thanks splunk/ansible-splunk) - name: Set admin access via seed when: splunk_first_run | bool block: - name: "Hash the password" command: "{{ splunk.exec }} hash-p... See more...
I have it this way (thanks splunk/ansible-splunk) - name: Set admin access via seed when: splunk_first_run | bool block: - name: "Hash the password" command: "{{ splunk.exec }} hash-passwd {{ splunk.password }}" register: hashed_pwd changed_when: hashed_pwd.rc == 0 become: yes become_user: "{{ splunk.user }}" no_log: "{{ hide_password }}" - name: "Generate user-seed.conf (Linux)" ini_file: owner: "{{ splunk.user }}" group: "{{ splunk.group }}" dest: "{{ splunk.home }}/etc/system/local/user-seed.conf" section: user_info option: "{{ item.opt }}" value: "{{ item.val }}" mode: 0644 with_items: - {opt: 'USERNAME', val: '{{ splunk.admin_user }}'} - {opt: 'HASHED_PASSWORD', val: '{{ hashed_pwd.stdout }}'} loop_control: label: "{{ item.opt }}" when: ansible_system is match("Linux") become: yes become_user: "{{ splunk.user }}" no_log: "{{ hide_password }}"  Then those user + pass information is in config file which are per environment etc. on git. All those secrets are saved by ansible-vault, so there is no passwords as plain text on your repository/inventory. You could have as many config files as you are needing. Usually one or more per environment and customer.
Hi @Eduardo.Rosa ! I don't think this is supported as of the moment. I'm using v24.3.1-1511 of Controller and it doesn't seem to have an option for Patch HTTP method. However you can share improveme... See more...
Hi @Eduardo.Rosa ! I don't think this is supported as of the moment. I'm using v24.3.1-1511 of Controller and it doesn't seem to have an option for Patch HTTP method. However you can share improvements and ideas with this specific item on our idea exchange. 
The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the integration of Artificial Intelligence (AI) and Machine Learning (ML) is revolutionizing the wa... See more...
The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the integration of Artificial Intelligence (AI) and Machine Learning (ML) is revolutionizing the way organizations maintain observability over their systems. Observability practices are enhanced by AI and ML to predict and prevent incidents before they occur, leading to significant improvements in reliability, availability, and customer experience. These technologies are applied across varying maturity stages, from gaining basic visibility to employing advanced predictive models that preempt potential issues. 1 The financial implications of downtime are profound, with each hour potentially costing organizations an average of $365,000, thus underscoring the critical need for efficient operational practices. This blog explores the transformative impact of AI and ML on observability within IT environments, demonstrating the benefits through various use cases and highlighting the substantial return on investment for organizations that adopt these practices. Harnessing AI and ML for Enhanced Observability   Organizations increasingly recognize the benefits of observability in enhancing reliability, availability and customer experience. The integration of Artificial Intelligence (AI) and Machine Learning (ML) into observability practices offers predictive capabilities, allowing for the anticipation and prevention of incidents before they occur. AI and ML are applied in three maturity stages for observability: from basic visibility without AI to proactive AI that establishes baselines for normal operations, and finally, to advanced predictive models that can preempt potential issues based on historical data. One of the roles of AI in observability is to provide a deeper understanding of 'normal' operational patterns and to direct attention to anomalies that require immediate action. This approach is instrumental in resolving issues faster, reducing downtime impact, and instilling confidence in maintaining customer-facing and internal services.     The complexity of modern technology stacks makes it challenging to discern operational health, a dilemma that AI and ML help to address by centralizing visibility and offering actionable insights. Features that support this include dynamic baselining, predictive modeling, and improving detection accuracy. These capabilities are encapsulated in a variety of products designed to simplify the adoption of AI and ML for organizations, regardless of their current maturity stage in observability. Understanding AIOps and Its Practical Applications   AIOps is the application of AI and  machine learning to operations. This includes anomaly detection, alert noise reduction, probable root cause, automation and remediation, and proactive prevention.. The importance of AIOps stems from its ability to predict and prevent potential IT issues before they impact customers, which is crucial for maintaining high availability and reliability of IT services. AIOps has a range of use cases, demonstrating its benefits across various scenarios. For instance, it aids in swiftly resolving downtime, enhancing detection efficiency, and reducing manual processing efforts. A practical example includes an organization that shifted from static to dynamic thresholding to improve alert accuracy, as static thresholds were either overwhelming during peak hours or missing anomalies during off hours. This shift led to massive improvements in detection efficiency. The monetary benefits of AIOps adoption are significant. Moreover, downtime not only incurs financial losses but also damages reputation, sometimes causing customers to switch to competitors. AIOps steps in as a vital tool to navigate the complexities of modern IT environments, providing a predictive approach to maintaining service performance and preventing issues. It achieves this by collecting data, defining Key Performance Indicators (KPIs), establishing baselines for typical performance, and building predictive models to anticipate and mitigate potential issues.   While the implementation of AIOps may seem challenging, tools and products are available to facilitate the transition, allowing organizations to start with simple use cases and gradually progress to more complex applications. Overall, AIOps offers a path to proactive IT operations management, enabling organizations to stay ahead of potential service performance issues and drive better business outcomes. The Challenge of Observability in the Age of Big Data   In today's digital landscape, organizations are grappling with the immense volumes of data generated by their IT environments. This surge in data presents a significant challenge in maintaining observability over their systems. As data volumes expand, the task of monitoring and managing the performance of IT services becomes increasingly complex. One of the critical issues faced in this scenario is the phenomenon of 'alert storms,' where the sheer quantity of alerts overwhelms the IT teams, making it difficult to pinpoint and troubleshoot performance issues effectively. The recent Splunk AI for Observability webinar revealed that organizations are indeed in the midst of what could be described as the 'perfect storm' of challenges, with many participants acknowledging struggles with growing data volumes, alert storms, and troubleshooting. Among these, too many alerts stood out as a particularly prominent issue, as it can obscure the root causes of performance degradation, leading to extended downtime and a scramble to restore services. The economic impact of such downtime is staggering. A survey published in the 'Digital Resilience Pays Off' report highlighted that, on average, each hour of downtime can cost organizations up to $365,000, not to mention the potential reputational damage that can arise from poor customer experiences. To combat these issues, organizations are turning to artificial intelligence (AI) to enhance their observability practices. AI is leveraged to predict or prevent incidents before they occur, and customers are segmented into three maturity stages based on their use of AI. The most advanced customers employ predictive models that can foresee and mitigate negative outcomes, whereas the less mature ones are yet to harness AI's full potential. In conclusion, the path to overcoming the challenges posed by growing data volumes lies in the strategic application of AI. The Evolution and Impact of AI and ML    Artificial Intelligence (AI) and Machine Learning (ML) have become integral components of Splunk products, offering significant advantages in detecting service performance issues. The integration of these technologies within Splunk's suite has a rich history, with nearly a decade of implementation. The profound impact of AI and ML is evident in the ability to predict or prevent incidents before they happen, enhancing the observability of systems. Organizations leveraging AI and ML in Splunk's offerings report substantial benefits, such as accelerated problem resolution and enhanced reliability of customer-facing and internal services. The evidence supporting the return on investment for those adopting observability practices is compelling, with statistics showing that each hour of downtime can cost an average of $365,000, highlighting the critical nature of maintaining operational efficiency. Successful AI and ML implementations have led to increased detection efficiency, reduced manual processing, and the identification of previously unknown scenarios within service performance. These advancements have been showcased through customer stories, such as the IG Group's transition from static thresholds to dynamic baselining, which has drastically improved detection efficiency. Another example is AIB's use of Splunk products to facilitate triage processes, leading to the discovery of an issue caused by unusual snowfall in Ireland. Lastly, StubHub's use of baseline models helped control application errors and uncover hidden issues. To streamline the adoption of AI and ML, Splunk ensures its products are accessible and supportive of users at any stage of their AI and ML journey, from simple use cases to complex predictive model deployments. The Splunk App for Anomaly Detection exemplifies this commitment, automating the detection of anomalies in key metrics and KPIs, thereby demonstrating Splunk's dedication to enhancing service performance through advanced technology. Enhancing Operational Efficiency with Anomaly Detection    The introduction of the Splunk App for Anomaly Detection marks a significant advancement in operational efficiency for organizations leveraging Splunk’s AI capabilities. A demonstration of this app in action reveals its capacity to effortlessly operationalize anomaly detectors for actionable alerting, thus simplifying the traditionally complex and technical process of anomaly detection. By automating the configuration for specific metrics or KPIs, the app streamlines the machine learning process to detect anomalies at ingest time, effectively removing barriers such as the need for complex SPL, statistical knowledge, and parameter tuning. The benefits of implementing anomaly detection are substantial and multifaceted. The demonstration highlights how this technology enables organizations to detect both isolated point anomalies and sustained anomaly intervals, accompanied by confidence scores to gauge the significance of the detected anomalies. Moreover, the app facilitates efficient detection by alerting users when anomalies with high confidence scores are found, and provides the SPL query generated by the app for further use within Splunk. Operational efficiency is further enhanced when anomaly detection is paired with predictive AI capabilities. Organizations can predict potential incidents before they occur, mitigating downtime and improving service reliability. This predictive approach is demonstrated by the Anomaly app's ability to identify anomalous behavior and to recommend appropriate actions in real-time, thus preventing potential service disruptions.     In conclusion, the Splunk App for Anomaly Detection exemplifies the intersection of AI and operational efficiency, providing a powerful tool for organizations to proactively manage and maintain the performance of their systems and services. Understanding Splunk's AI Principles and the AI Assistant   Splunk integrates AI into its observability suite, offering domain-specific AI capabilities that enhance the efficiency of AI-driven processes. The involvement of humans remains crucial, ensuring that AI assists rather than replaces human decision-making. A notable innovation is the introduction of the Splunk AI Assistant for SPL, which simplifies the creation of SPL queries and their understanding. This assistant, currently in public preview, promises to unlock the full potential of SPL-powered Splunk products. Statistics underline the importance of AI in observability, demonstrating how AI applications in Splunk significantly reduce downtime costs, which can average $365,000 per hour. With complex technology stacks, AI helps organizations predict and prevent incidents, using models to forewarn about potential issues and enabling quick resolution. The AI-driven capabilities are designed to be open and extensible, allowing users to customize models or employ their own, maintaining Splunk's versatile problem-solving essence. The Splunk AI Assistant aims to be a scalable aid across the ecosystem, with future enhancements like an AI assistant for observability cloud and an improved adaptive thresholding experience. In essence, Splunk's AI principles and the AI Assistant offer a comprehensive, user-friendly AI integration that fosters proactive, informed, and efficient observability practices. Enhancing Security and Observability with AI and Unified Platforms   The integration of Artificial Intelligence (AI) into observability practices significantly accelerates the ability to detect, investigate, and respond to incidents. AI's role in enhancing security and observability is pivotal, particularly in predicting or preventing incidents before they occur. Organizations are adopting AI to gain insights on what normal performance looks like in their environments and to detect deviations that may indicate emerging issues. With the growth of data volumes and the complexity of technology stacks, AI becomes an indispensable tool in managing alert storms and troubleshooting performance issues. A survey by Splunk has found that each hour of downtime can cost organizations an average of $365,000, underscoring the financial impact of operational disruptions beyond the reputational damage it may cause to customer-facing services. The comprehensive nature of the Splunk platform caters to this need, with its decade-long incorporation of AI and machine learning to support a wide range of use cases. Furthermore, the importance of a unified platform cannot be overstated. Splunk's unified platform serves as a cohesive foundation for SecOps, ITOps, and DevOps teams, providing solutions that are tailored to their specific requirements while reaping the benefits of a holistic approach. This unified approach ensures that these teams can operate more efficiently, with better visibility and tools for rapid response, thereby reducing mean time to resolution (MTTR) and mitigating the risk of service performance issues. AIOps: A Mature Approach to Observability   In the increasingly complex IT landscape, AIOps emerges as a mature approach to observability, playing a pivotal role in enhancing incident response and operational efficiency. This approach is fostered by the need to predict or prevent incidents before they occur, leading to continuous improvement and predictive analytics capabilities within IT operations. The integration of AIOps has demonstrated a significant impact, particularly in the realm of incident management. Organizations that embrace AIOps have observed a faster resolution of problems, resulting in improved availability and reliability of services. Notably, customers leveraging AIOps have reported the ability to fix issues more swiftly and confidently maintain customer-facing and internal services. A mature AIOps strategy encompasses various stages of implementation, ranging from basic visibility into IT environments to advanced predictive models. These models can anticipate potential negative outcomes based on historical data, enabling preemptive remediation efforts. The statistical evidence supporting AIOps is compelling, with reports indicating that each hour of downtime can cost an average of $365,000, highlighting the financial incentives to adopt such strategies. Moreover, AIOps facilitates the identification of previously unknown scenarios, thereby driving innovation and informed decision-making. The journey toward AIOps maturity not only increases detection efficiency but also reduces manual processing and uncovers new insights, all of which contribute to the overarching goal of achieving digital resilience and operational excellence.     In conclusion, the integration of AI and ML into observability practices is recognized as a transformative force in IT operations. It is demonstrated that these technologies significantly enhance the ability to predict and prevent incidents, leading to improved service reliability and customer experience. The financial benefits are underscored by the high cost associated with downtime, encouraging organizations to adopt AI and ML to maintain operational efficiency. Through various use cases, the effectiveness of AI-driven solutions in addressing complex technology stack challenges is illustrated, showing a substantial return on investment. A mature approach to observability, encompassing AIOps, is presented as essential for ongoing improvement and innovation within organizations. It is concluded that AI and ML are indispensable in the contemporary digital landscape for driving business resilience and efficiency.  1 From Digital Resilience Pays Of (p.7) by  Splunk © 2023 Splunk Inc. 
Also, our SMEs don't all patch their servers on the same day, but they usually patch similar servers on the same day. So Monday, they might patch IDS servers, and Wednesday they might scan the vulner... See more...
Also, our SMEs don't all patch their servers on the same day, but they usually patch similar servers on the same day. So Monday, they might patch IDS servers, and Wednesday they might scan the vulnerability scanners and the following week they might patch all the application servers, etc.  So the installed dates are frequently different  and sometime versions are different between all our linux hosts but are consistent between the servers of the same type (IDS, Scanner, Application, etc.)
Also, our SMEs don't all patch their servers on the same day, but they usually patch similar servers on the same day. So Monday, they might patch IDS servers, and Wednesday they might scan the vulner... See more...
Also, our SMEs don't all patch their servers on the same day, but they usually patch similar servers on the same day. So Monday, they might patch IDS servers, and Wednesday they might scan the vulnerability scanners and the following week they might patch all the application servers, etc.
How do you wish them to "be combined"?
The link between the two searches would be our monthly list of installed packages, after patching we gather the current list of installed packages and ingest the data into Splunk. We would like to co... See more...
The link between the two searches would be our monthly list of installed packages, after patching we gather the current list of installed packages and ingest the data into Splunk. We would like to compare the list from the current month to the lists from previous months, because not all packages have an update/patch each month. So for kernel as an example, there are frequent updates/patches and usually changes every month. But, for less frequently update/patched packages we might need to compare back two or more months. So I would want to compare the current installed packages with the last two or even as far back as six months or a year. I thought if I "joined" the list of previous installed packages that had been deduped or stats latest(version) AS previous_version, latest(_time) AS previous_installed_date by package, I could capture the last version and installed date of each package. search 1 would have the list of the current packages - package, installed date, version search 2 would have the list of the last installed date and last version of all previously installed packages with different field names for installed date and version the join would join the two lists by package name output would be package, version, installed date, last version, last installed date
An example would be the linux kernel, for instance if this month's patching process a new kernel patch was applied, we would like to find the last kernel version that was installed, the month before,... See more...
An example would be the linux kernel, for instance if this month's patching process a new kernel patch was applied, we would like to find the last kernel version that was installed, the month before, two months before or even earlier if that was the case. Kernel version change fairly regularly, but some of the other linux packages might change/update a little less frequently. After we patch we capture the list of installed packages and ingest the data into Splunk. So every month we have the data of the current installed packages, for compliance reasons, we need to verify what packages were updated during our patching process. So we are trying to compare the latest installed packages list, with the installed package lists from previous months. Our output would be something like this: package     current version    install date    previous version    previous install date kernel    ssh python glibc etc...
Hello, I have a standalone Splunk Enterprise 9.1.3 instance with some DCs and servers connected to it using Forwarder Management console. At the moment I have 2 server classes configured, 1 for the... See more...
Hello, I have a standalone Splunk Enterprise 9.1.3 instance with some DCs and servers connected to it using Forwarder Management console. At the moment I have 2 server classes configured, 1 for the DCs and the other one for the servers. The server class for the DCs includes only the inputs.conf file for Windows logs: [WinEventLog://Security] disabled = 0 index = myindex followTail=true start_from = oldest current_only = 0 evt_resolve_ad_obj = 1 checkpointInterval = 5 whitelist = 4624,4634,4625,4728,4729 renderXml=false Moreover, in the Splunk Enterprise I configured 2 transforms for splitting the logs in two separeted indexes, like this: props.conf: [WinEventLog:Security] TRANSFORMS-security = rewrite_ad_group_management, rewrite_index_adm transforms.conf: [rewrite_ad_group_management] REGEX = EventCode=(4728|4729) DEST_KEY = _MetaData:Index FORMAT = index1 [rewrite_index_adm] REGEX = Account Name:\s+.*\.adm DEST_KEY = _MetaData:Index FORMAT = index2 In particular, the goal is to forward the authentication events (4624,4634,4625) for only admin users (Account Name:\s+.*\.adm) in index2 and only EventCode 4728 and 4729 in index1, and the events that not match none transform should remain in myindex. At the moment the first transform is not working, so I'm receiving Events 4728 and 4729 in index2, am I missing something or there is a better logic to do that? I tried to combine also 4624,4634,4625 and Account Name:\s+.*\.adm with  (?ms)EventCode=(4624|4634|4625)\X*Account Name:\s+.*\.adm Thanks in advance
Hi, @aasserhifni , surely there's a misunderstanding: a SH can be managed by a Deployer only in a SHCluster, a Deployer cannot manage a stand-alone SH. Probably you mean a Deployment Server, that's... See more...
Hi, @aasserhifni , surely there's a misunderstanding: a SH can be managed by a Deployer only in a SHCluster, a Deployer cannot manage a stand-alone SH. Probably you mean a Deployment Server, that's one of the checks I hinted. If your SH is managed by a Deployment Server, you have only to remove the App from the ServerClass where the SH is present. Ciao. Giuseppe