All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hello, I'm not finding info on the limits within Splunk's data rebalancing. Some context, I have ~40 indexers and stood up 8 new ones. The 40 old ones had an avg of ~150k buckets each. At some poi... See more...
Hello, I'm not finding info on the limits within Splunk's data rebalancing. Some context, I have ~40 indexers and stood up 8 new ones. The 40 old ones had an avg of ~150k buckets each. At some point the rebalance reported that it was completed (above the .9 threshold) even though there were only ~40k buckets on the new indexers. When I kicked off a second rebalance, it started from 20% again and continued rebalancing because the new indexers were NOT space limited on the smartstore caches yet. The timeout was set to 11 hours and the first one finished in ~4. The master did not restart during this balancing. Can anyone shed some more light on why the first rebalance died? Like, is there a 350k bucket limit per rebalance or something?
Hi @Priya70  This behaviour, where a search completes but the panel fails to render and the progress bar stays at 0%, often indicates a client-side rendering issue or a problem with the data format ... See more...
Hi @Priya70  This behaviour, where a search completes but the panel fails to render and the progress bar stays at 0%, often indicates a client-side rendering issue or a problem with the data format or volume being passed to the visualization component in the browser. Here are steps to troubleshoot: Check Browser Developer Console: Open your browser's developer tools (usually F12), go to the "Console" tab, and look for any JavaScript errors when the problematic panel is attempting to load or after the search completes. This is the most common place to find clues about rendering failures. Simplify the Search/Visualization: Temporarily simplify the search query for the affected panel. Try changing the visualization type to a simple table. If the table renders correctly, the issue is likely with the specific visualization type or its configuration. If even a simple table fails, the issue might be with the data itself or a more fundamental client-side problem. Verify Data Format: Ensure the fields and data types returned by your search match what the chosen visualization expects. For example, a timechart requires _time and numerical fields. Consider Data Volume: While the search completes, rendering a very large number of data points or complex structures can sometimes overwhelm the browser or the visualization library, leading to rendering failure. Try adding | head 100 to your search to limit results and see if it renders. Test in Different Browser/Incognito Mode: Rule out browser-specific issues, extensions, or cached data by testing the dashboard in a different browser or an incognito/private browsing window.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@Raj_Splunk_Ing  Modify Your Python Script in Alteryx: You need to find the part of your Python script where you build the URL or the parameters for the HTTP request to Splunk. You will then add a ... See more...
@Raj_Splunk_Ing  Modify Your Python Script in Alteryx: You need to find the part of your Python script where you build the URL or the parameters for the HTTP request to Splunk. You will then add a tz parameter to that request, setting its value to the time zone string you found in your splunk ui timezone. Eg: params = { 'search': spl_query, 'output_mode': 'csv', 'tz': splunk_ui_timezone } url adding tz parameter https://server/services/search/jobs/export?search=search%20index%3Dcfs_apiconnect_102212%20%20%20%0Asourcetype%3D%22cfs_apigee_102212_st%22%20%20%0Aearliest%3D-1d%40d%20latest%3D%40d%20%0Aorganization%20IN%20(%22ccb-na%22%2C%22ccb-na-ext%22)%20%0AclientId%3D%22AMZ%22%20%0Astatus_code%3D200%0Aenvironment%3D%22XYZ-uat03%22%0A%7C%20table%20%20_time%2CclientId%2Corganization%2Cenvironment%2CproxyBasePath%2Capi_name&tz=America%2FChicago&output_mode=csv Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
@tangtangtang12  NEAPs are primarily triggered when an episode's severity changes (e.g., Normal -> Critical, or Critical -> High) or when a new notable event matches the policy. The most robust and... See more...
@tangtangtang12  NEAPs are primarily triggered when an episode's severity changes (e.g., Normal -> Critical, or Critical -> High) or when a new notable event matches the policy. The most robust and common way to achieve this involves using a Saved Search You need a search that identifies services currently in a critical or high state. Eg: | itsi_get_service_health | search service_health_score > 0 AND service_health_score < 60 /* Or whatever your critical/high thresholds are. Typically: Critical < 40, High < 60 (or similar) Adjust based on your ITSI configuration. */ | rename title AS itsi_service_name | fields itsi_service_name, service_health_score, severity_label | eval alert_message = "ITSI Service " + itsi_service_name + " is still " + severity_label + " (Health: " + service_health_score + ")." Then save as a Saved Search/Alert with your desired schedule. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
Thank you so much for the prompt answer!
@danielbb  _time is being set by the Tenable Add-on itself, using a timestamp field from the Tenable API response (e.g., last_seen, last_found). DATETIME_CONFIG = NONE in props.conf for tenable i... See more...
@danielbb  _time is being set by the Tenable Add-on itself, using a timestamp field from the Tenable API response (e.g., last_seen, last_found). DATETIME_CONFIG = NONE in props.conf for tenable is intentional to prevent Splunk from trying to re-parse _time from the event's raw data and potentially overriding the TA's carefully chosen timestamp Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
@Lien  You're right, connecting Splunk Cloud directly to an on-premise Active Directory via LDAP is generally not the recommended or straightforward approach, and SAML is highly preferred. Why SAML... See more...
@Lien  You're right, connecting Splunk Cloud directly to an on-premise Active Directory via LDAP is generally not the recommended or straightforward approach, and SAML is highly preferred. Why SAML is Better for Splunk Cloud: Enhanced Security: -Your AD is not directly exposed to the internet for Splunk Cloud. -Authentication happens at your IdP. Splunk Cloud trusts the assertion from your IdP. -Easier to enforce Multi-Factor Authentication (MFA) via your IdP. Standardized Integration: SAML is a web browser SSO standard. It's well-understood and robust. Centralized Identity Management: Leverages your existing identity management infrastructure. No Direct Network Dependency:Splunk Cloud doesn't need a persistent network connection to your AD for authentication transactions. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
@mchoudhary  Multiple appends: Fundamentally, this is the main performance killer. You're running 6 distinct "heavy" searches. Main performance killer is on the transaction in EDR: The transaction ... See more...
@mchoudhary  Multiple appends: Fundamentally, this is the main performance killer. You're running 6 distinct "heavy" searches. Main performance killer is on the transaction in EDR: The transaction "event.DetectId" command is notoriously resource-intensive, especially over a 6-month period. It should be avoided if at all possible, or its scope drastically limited   Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
I am thinking about which way is better to use LDAP(AD) or SAML for authentication of Splunk Cloud. Unlike Splunk standalone, the cloud version looks like a little tricky. I read some document that... See more...
I am thinking about which way is better to use LDAP(AD) or SAML for authentication of Splunk Cloud. Unlike Splunk standalone, the cloud version looks like a little tricky. I read some document that Splunk Cloud is not recommend to connect to AD- LDAP directly somewhere. But I could not find where they are. I am trying to connect LDAP from Splunk Cloud, but always got error and there were very few inforamtion showing in splunkd.log Can someone let me know if the direct connect to AD LDAP from Spunk cloud is recommended or not? Also if there is any trouble shooting tool can easily built the connection?
Do you have any base searches? Are any of the panels driven by saved searches?  
@Priya70  If the search returns a large dataset or the panel uses a complex visualization rendering might silently fail or else Memory or CPU limitations in the browser can cause rendering to hang, ... See more...
@Priya70  If the search returns a large dataset or the panel uses a complex visualization rendering might silently fail or else Memory or CPU limitations in the browser can cause rendering to hang, especially with multiple panels loading simultaneously. If multiple panels use similar base searches, consider using a base search with postProcess to reduce load. Can you please paste your dashboard XML to identify the issue? I’ll take a look..
Hi,
How did you figure out that the problem was with that firewall search? You are doing lots of appends, which is always going to slow down any search, as you are pushing everything to the search head,... See more...
How did you figure out that the problem was with that firewall search? You are doing lots of appends, which is always going to slow down any search, as you are pushing everything to the search head, but also you are doing an unbounded 6 month transaction statement, which again is going to be pretty slow and potentially unpredictable. What are your data volumes for each individual search? The separate times for your 6 individual searches will not sum up to give you the expected cost of the overall search and I would not actually expect the tstats searches per se to be the source of any performance problem. If you are searching summaries only from accelerated datamodels, they should be the least of your worries. Your CrowdStrike search is getting ALL the data for 6 months and running transaction on that, which if you have any kind of volume there, is unlikely to be reliable, because transaction will silently drop results when it hits limits.  
@danielbb I wasn't able to find anything; we need to build a brand new one. You should be able to quickly build one using Dashboard examples. If you have ondemand credits on your account entitlement... See more...
@danielbb I wasn't able to find anything; we need to build a brand new one. You should be able to quickly build one using Dashboard examples. If you have ondemand credits on your account entitlement, you can also leverage PS expert service to build dashboards; they can shoulder surf to get you started. Assets Inventory Example:   index=<your_tenable_index> sourcetype=tenable:io:assets| eval ip=mvindex(ipv4, 0) | stats count by hostname, ip, os, last_seen, tags Plugin Overview Example:   index=<your_tenable_index> sourcetype=tenable:io:plugin | stats count by plugin_name, plugin_id, family Audit Log Events Example:   index=<your_tenable_index> sourcetype=tenable:io:audit_logs | timechart count by action
1. As @richgalloway mentioned, a (typically small but it realy depends on the input) lag between _time and _indextime is a normal state. Or at least on its own it doesn't mean that something is wrong... See more...
1. As @richgalloway mentioned, a (typically small but it realy depends on the input) lag between _time and _indextime is a normal state. Or at least on its own it doesn't mean that something is wrong. 2. DATETIME_CONFIG=none explicitly disables timestamp recognition. Are you sure it is what you want? 3. If there is a difference between the timestamp included in the raw event and the timestamp stored in the _time field, the data is not properly onboarded. Tenable.io is a cloud service so I suppose there is some modular input which pulls the data from the cloud and pushes them to Splunk. But I have no idea whether the timestamps should be parsed by the input itself and fed "as is" to Splunk or if the data should be parsed in Splunk. Infortunately, it's a third party add-on so there can be completely everything happening inside...
Thank you @richgalloway    My question is, why an app sets something like? [tenable:io:vuln] DATETIME_CONFIG = NONE   That's what this tenable TA does, I don't get it.
DATETIME_CONFIG = CURRENT != NONE  Current set the timestamp from the Aggregation queue time.  None in this instance sets the timestamp from the time handed over to Splunk by the modular input sc... See more...
DATETIME_CONFIG = CURRENT != NONE  Current set the timestamp from the Aggregation queue time.  None in this instance sets the timestamp from the time handed over to Splunk by the modular input script.  Splunk then still needs to send the data to an indexer, which is where the _indextime will be set. Yes the data is cooked and time set on the HeavyForwarder but note the _indextime is NOT.  An easy way to see this in action is to look at any of your data being ingested by DBConnect with _time being set to Current. The _indextime will usually be negative,  but every once in a while you'll see it jump to a few seconds usually due to a blocked output queue. And of course any difference between the HeavyForwarder and Indexer time will of course cause times to be off as well. 
Are you sure this is your literal search? Because you cannot pipe to a tstats command unless it's with prestats=t append=t. Also, what does your security_content_summariesonly macro expand to? Also... See more...
Are you sure this is your literal search? Because you cannot pipe to a tstats command unless it's with prestats=t append=t. Also, what does your security_content_summariesonly macro expand to? Also also - you're appending two "full" searches. Are you sure you're not hitting subsearch limits? And back to the point - that's what job details and job log are for - see the timeline, see where Splunk spends its time. Check the scanned results vs. returned results...
Currently, we receive a single email alert via Notable Event Aggregation Policies (NEAP) whenever our ITSI services transition from normal to high or critical. However, we need an automated process t... See more...
Currently, we receive a single email alert via Notable Event Aggregation Policies (NEAP) whenever our ITSI services transition from normal to high or critical. However, we need an automated process that sends recurring email alerts every 5 minutes if the service remains degraded and hasn't reverted back to normal. From my research, many forums and documentation suggest achieving this through Correlation Searches. However, since we rely on KPI alerting, and none of our Correlation Searches (even the out-of-the-box ones) seem to function properly, this approach hasn't worked for us... Given the critical nature of the services we monitor, we’re seeking guidance on setting up recurring alerts using NEAPs or any other reliable method within Splunk ITSI. Any assistance or insights on how to configure this would be greatly appreciated.
Hi everyone! I am working on building a dashboard which captures all the firewall, Web proxy, EDR, WAF, Email, DLP blocked for last 6 months in a table format which should look like this -  I... See more...
Hi everyone! I am working on building a dashboard which captures all the firewall, Web proxy, EDR, WAF, Email, DLP blocked for last 6 months in a table format which should look like this -  I am able to write the query which will give me the count for each parameter and then I append all the single query into one which is making the final query run slower and taking forever to complete. Here is the final query- | tstats `security_content_summariesonly` count as Blocked from datamodel=Network_Traffic where sourcetype IN ("cp_log", "cisco:asa", "pan:traffic") AND All_Traffic.action="blocked" earliest=-6mon@mon latest=now by _time | eval Source="Firewall" | tstats `security_content_summariesonly` count as Blocked from datamodel=Web where sourcetype IN ("alertlogic:waf","aemcdn","aws:*","azure:firewall:*") AND Web.action="block" earliest=-6mon latest=now by _time | eval Source="WAF" | append [search index=zscaler* action=blocked sourcetype="zscalernss-web" earliest=-6mon@mon latest=now | bin _time span=1mon | stats count as Blocked by _time | eval Source="Web Proxy"] | append [| tstats summariesonly=false dc(Message_Log.msg.header.message-id) as Blocked from datamodel=pps_ondemand where (Message_Log.filter.routeDirection="inbound") AND (Message_Log.filter.disposition="discard" OR Message_Log.filter.disposition="reject" OR Message_Log.filter.quarantine.folder="Spam*") earliest=-6mon@mon latest=now by _time | eval Source="Email"] | append [search index=crowdstrike-hc sourcetype="CrowdStrike:Event:Streams:JSON" "metadata.eventType"=DetectionSummaryEvent metadata.customerIDString=* earliest=-6mon@mon latest=now | bin _time span=1mon | transaction "event.DetectId" | search action=blocked NOT action=allowed | stats dc(event.DetectId) as Blocked by _time | eval Source="EDR"] | append [search index=forcepoint_dlp sourcetype IN ("forcepoint:dlp","forcepoint:dlp:csv") action="blocked" earliest=-6mon@mon latest=now | bin _time span=1mon | stats count(action) as Blocked by _time | eval Source="DLP"] | eval MonthNum=strftime(_time, "%Y-%m"), MonthName=strftime(_time, "%b") | stats sum(Blocked) as Blocked by Source MonthNum MonthName | xyseries Source MonthName Blocked | addinfo | table Source [| makeresults count=7 | streamstats count as month_offset | eval start_epoch=relative_time(now(),"-6mon@mon"), end_epoch=now() | eval start_month=strftime(start_epoch, "%Y-%m-01") | eval month_epoch = relative_time(strptime(start_month, "%Y-%m-%d"), "+" . (month_offset-1) . "mon") | where month_epoch <= end_epoch | eval month=strftime(month_epoch, "%b") | stats list(month) as search ]   I figured out the issue is with the firewall query- | tstats `security_content_summariesonly` count as Blocked from datamodel=Network_Traffic where sourcetype IN ("cp_log", "cisco:asa", "pan:traffic") AND All_Traffic.action="blocked" earliest=-6mon@mon latest=now by _time | eval Source="Firewall" Can someone guide me how to fix this issue. I have been stuck in this issue from 2 weeks