Adding to sherm77's response, you can set the refresh type and refresh interval for individual panels using the <search> element. The <refreshType> element is either 'interval' or 'delay'. A value of 'interval' means a <refresh> value of 30 seconds causes a refresh every 30 seconds regardless of the state of the search. A value of 'delay' means the 30 second countdown only begins once the search has completed. Example Usage: <form> <row> <panel> <table> <search> <query> ... </query> <earliest>-60m@m</earliest> <latest>now</latest> <refresh>60</refresh> <refreshType>delay</refreshType> </search> </table> </panel> </row> </form>   ... View more

First, are you using Splunk Enterprise version 8.x? Earlier versions are not compatible with the latest ES version of 6.6.x: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix Next, check the post-upgrade documentation to confirm you've completed all the required steps to have ES working correctly. Also verify that your previous notable events are present, as these may need to be restored from your KV store backup given the large gap between Splunk ES versions. Create a manual notable event, and verify that this appears in the incident review page. This will ensure that Enterprise Security is (for the most part) configured correctly. If required, check index=_internal sourcetype=splunkd source=*splunkd.log " ERROR " OR " WARN ". You may want to check other log sources as well, but this will point to the most significant problems. Confirm that correlation searches have run, and verify that they are generating events that result in creation of notables. The upgrade from 4.x to 6.x is quite significant and much may have changed in the SPL of these alerts. E.g. they may have referred to indexes and sourcetypes previously, but now refer to datamodels. If necessary, create your own enabled correlation search with a notable event action and verify that this appears on the incident review page. If the searches are running, and the notable event index is populating, but incident review is empty, check to see if notable events are being created in index=notable, and if so, check the `notable` and `incident_review` macros to determine if any errors are occurring during enrichment. This could indicate a problem with the KV store lookups or the underlying MongoD service that it runs on. Let us know how you go. ... View more

I've seen this issue on Linux hosts with Splunk 7.3.x, where Splunk admin roles were assigned via RBAC using AD groups. There was an issue where the AD group membership was not being correctly identified on Linux, which resulted in the user still showing as admin in Splunk, but not having Splunk admin privileges. The issue was fixed during a Splunk upgrade, but you could also try the following if your situation is similar: - Running systemctl restart sssd - Open settings -> authentication methods -> reload - Use the CLI or make an API call to reload the authentication: ./splunk reload auth curl -k -u admin:changeme https://splunkserver:8089/services/authentication/providers/services/_reload Also note that SAML can take up to 10 minutes to fully populate the list of analysts that can be assigned to a notable event. It may be worth waiting patiently for this list to fully update. Alternatively, assign the admin role locally on the ES SH to rule out other authentication technologies interfering with the required permissions. Reference: https://docs.splunk.com/Documentation/ES/6.6.2/User/Triagenotableevents#Assign_notables_to_owners   ... View more

Good news! The threat intelligence framework does all the hard work for you. The following link provides an overview of the process, but the main point is that there's a correlation search called "Threat Activity Detected". https://dev.splunk.com/enterprise/docs/devtools/enterprisesecurity/threatintelligenceframework/ To configure alerts based on threat intelligence, open Splunk Enterprise Security, click on Content Management, and select a Type of "correlation search". Locate the "Threat Activity Detected" correlation search and enable the search. By default, the search will run at 10 minutes past each hour and look for threats over the last 65 minutes. Once detected, threat matches will not be reported for another 24 hours. ... View more

  You are using the following command curl -k https://10.1.10.20:8088/services/collector/ack?channel=FE0ECFAD-13D5-401B-847D-77833BD77132 -H "Authorization: Splunk 9cedcd53-b32d-43ba-9cb6-25a211c720bc" -d "{"acks":"0"}" The format you should be using per https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/AboutHECIDXAck is as follows: curl https://mysplunk.com/services/collector?channel=FE0ECFAD-13D5-401B-847D-77833BD77131 -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d '<data>' -v  You are referencing collector/ack?channel, but should be referencing collector?channel. ... View more

I wouldn't recommend it. When indexer acknowledgement is enabled, the purpose of the channel is to prevent a fast client from impacting the performance of a slow client. Reference: About HEC Indexer Acknowledgement  ... View more

To delete the config for the collection, rather than the records within it, use: curl -k -u admin:changeme -X DELETE https://<host>:8089/servicesNS/nobody/<app>/storage/collections/config/<collection> Deleting the transform lookup configuration may still work, but this is the solution currently provided in the Splunk docs at https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/usetherestapitomanagekv/ ... View more

You can calculate based on the total using eventstats. E.g. | eventstats sum(P50) as P50sum sum(P90) as P90sum New columns of P50sum and P90sum will be added to each row. These will contain the respective values of 100 and 200 that you currently have in the totals row. Marker P50 P50sum P90 P90sum Point 1 Duration 10 100 20 200 Point 2 Duration 40 100 100 200 Point 3 Duration 50 100 80 200 From here, you can run eval and fieldformat commands to calculate based on the two row fields: | eval P50dec = P50/P50sum | eval P90dec = P90/P90sum | fieldformat P50pc = printf("%%.1f", P50dec*100) | fieldformat P90pc = printf("%%.1f", P90dec*100) The eval commands create exact decimal values, while fieldformat formats these as percentage values to two decimal places without changing the underlying data. If you want, you can use the same field names for both. Finally, table the output to display the fields you want in the desired order. Marker P50 P50sum P50pc P90 P90sum P90pc -------------------------------------------------------------------------------------- Point 1 Duration 10 100 10.0% 20 200 10.0% Point 2 Duration 40 100 40.0% 100 200 50.0% Point 3 Duration 50 100 50.0% 80 200 40.0% If you want running totals for each checkpoint, you could use streamstats rather than eventstats instead. ... View more

You could append | outputlookup to store the results with the current datetime (e.g. cs_status). You can then either: Use | lookup (prior to | outputlookup) to compare current REST results with most recent stored results Create a separate search (assuming daily lookup of cs_status and time field of cs_time) with: | inputlookup cs_status | search cs_time > relative_time(now(), "-48h") | stats count, values(cs_time) as cs_time by Name disabled | where count=1 If the search is created/deleted, or the disabled state changes, the count should equal one. If you don't want the search creation and enable events, some additional logic is required to compare the values of all three fields (Name, disabled and cs_time). ... View more

You mentioned you have other working searches. Are there any differences between these and the search above that is not working? I.e. are you able to create other correlation searches based on these differences to replicate the issue? If you recreate the correlation search, does the problem continue? Incident review functionality is typically open to anyone with the ess_analyst role, but it may be helpful to see if the same issue occurs for an admin role (if you're not already logged in as an admin). In ES Content Management, is the correlation search listed and enabled, with a notable event action? Are there any conflicting alerts with a notable event action (in searches, reports & alerts)? Are there any logs that might provide further information? See https://community.splunk.com/t5/Splunk-Enterprise-Security/How-do-I-determine-why-a-Correlation-Search-isn-t-creating-a/m-p/370277 for details. I.e. can the correlation search and notable event be traced from start to finish without error? You can also check the following for other potential issues at the time of the notable firing and the time you perform your search. You may want to remove the source file to expand the search, and optionally filter by other keywords relevant to your search: index=_internal source=*splunkd.log sourcetype=splunkd " ERROR " OR " WARN "   ... View more

I realise this is a post from 2016, but I noticed that the <input> uses a <default> value rather than the recommended <initial> value per the Splunk docs (for the current version of 8.2.3), which may help others while troubleshooting this or similar issues on newer Splunk versions. https://docs.splunk.com/Documentation/Splunk/8.2.3/Viz/FormEditor#Configure_input_value_handling ... View more

The good news is that you can definitely add your own fields. However, it may be best to make use of the existing fields as shown in the link below. I suggest this because ES has changed the fields for these lookups in the past, and changes to the default may complicate migrations or upgrades in the future. https://docs.splunk.com/Documentation/ES/latest/Admin/Formatassetoridentitylist#Identity_lookup_header Fields that might meet your need include category, watchList & endDate (as you mentioned). You can then create your own identity list and optionally prioritise it to ensure that any risk ratings apply to the values you've entered rather than those obtained through other sources such as LDAP/AD. If you're not worried about data migration, or if you're comfortable modifying a migration script to include your additional source field if necessary, you can follow the steps below to add or modify identity fields for the current version. Be aware that any existing lookups will need to have the additional field name and column added for existing entries to prevent errors from occurring when these lookups are queried. https://docs.splunk.com/Documentation/ES/6.6.2/Admin/Identitysettings Hope this helps. ... View more

Correlation searches in savedsearches.conf have a search name, as well as a correlation search name as an attribute within the stanza, which from memory is the display name of the correlation search. The correlation keyword search may be matching on the actual search name rather than what is displayed in the results from the free text keyword search. Alternatively, it may be due to minor breakers in the target string. I.e. if there's a special character in the target string, it may need to be included in the search in order to match. Let us know if you're able to match based on the alternate search name or different keywords. ... View more

No, there is no dependency between Splunk Security Essentials (SSE) and Enterprise Security Content Updates (ESCU), but you should definitely update the ESCU frequently as it provides monthly updates to analytics, analytic stories, and associated content. However, SSE does specify recommended add-ons within the app itself, stated as follows from the link below: "On the Security Content page, add-ons that you need to configure to make a detection work are now recommended and linked to from the detection. " https://docs.splunk.com/Documentation/SSE/latest/ReleaseNotes/Enhancements   ... View more

I've dug into this one a bit, and I believe it could be due to the way Splunk uses XML to handle the visualisations. Have you tried using stats instead of chart or table, and then formatting the visualisation as a chart using the GUI? The XML for an area chart should look something like the following, for example: <panel> <chart><title>...</title> <search><query>...</query>...</search> <option name="charting.chart">area</option> </chart> </panel> I.e. the chart XML is independent from the results returned by the search. I've worked almost exclusively with on-premises instances, so I can't speak to the accessibility of the _internal index in Splunk cloud, though pressing F12 in your browser to display the console may hint at any other problems that could be occurring here. Let me know how you go. ... View more

Since you're new it's probably best to start with an overview at the link below, noting that the threat framework does the work in the background for you to generate notables for your security data: https://www.youtube.com/watch?v=NJT-fE35eaY Splunk allows you to trigger notable events based on threat intel information. Start by configuring the threat intel following the directions here: https://docs.splunk.com/Documentation/ES/6.6.2/Admin/Addthreatintel Nearly every threat intel source will reference the EICAR test file, so I would recommend downloading that after setting up the threat intel to verify that the notable fires correctly. https://www.eicar.org/?page_id=3950 Even if you planned on creating your own lookups, these could be integrated into ES as a new threat intel source to be managed and prioritised along with existing threat intel sources. It's definitely not a case of 'one or the other' If you wanted to set up a custom search despite what ES provides, and you decided not to use guided mode for the correlation search, my recommendation would be to start with the "Network_Traffic" datamodel to search on the desired dataset, and progress to using tstats to form efficient searches. As long the datamodel references your index appropriately, you're good to go. If you have everything configured, but don't seem to be getting the results you expect, please provide some additional detail on what you have configured, what tests you've performed, and what results you received. ... View more

To start with, this document is your new best friend: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix Out of Splunk Enterprise and Enterprise Security, you'll want to upgrade Splunk Enterprise first, with a caveat in the section below... If you're currently running Splunk ES 6.4 with Splunk 7.x.x, you'll notice that this isn't considered compatible, and you may run into problems during the upgrade. Splunk recommends updating Splunk Enterprise and ES in the same change window, but in your case, you should focus on getting Splunk Enterprise updated to a compatible state and supported edition before progressing further. Normally, it would also matter whether you're running ES 6.4.0 or 6.4.1, as Splunk 8.2.2 is compatible with 6.4.1 but not 6.4.0. However, again, the lack of compatibility could cause problems so you should upgrade to 8.1 first, and then 8.2 per the documentation below: https://docs.splunk.com/Documentation/Splunk/8.2.2/Installation/HowtoupgradeSplunk Next, read the instructions, and here comes the caveat. The biggest change to Splunk 8 is the use of Python 3, and there is an app that lets you verify app compatibility in the new Splunk version. If your apps don't support Python 3 they may stop working. This is particularly important if Splunk is integrated with older third party applications, as some apps contain APIs that are application-version-specific. It's also relevant for any custom apps that have been written, or for apps that are no longer supported or have (still) not been upgraded for Python 3 compatibility (yes, there are a few). In short, read the manual and know your environment before proceeding. https://docs.splunk.com/Documentation/Splunk/8.0.2/Installation/AboutupgradingREADTHISFIRST In terms of what infrastructure components to upgrade first, you should take a look at this resource, and it should form the basis for your upgrade plan: https://community.splunk.com/t5/Installation/What-s-the-order-of-operations-for-upgrading-Splunk-Enterprise/m-p/408003 Basically, upgrade management components first, then search heads, then indexers. Note the verification between each management component upgrade. Ensure you have snapshots available and backups of data where appropriate, including backups of the KV store on the search head, as this is crucial to Enterprise Security. Once you're up to Splunk 8.1.x on all servers, it's fairly smooth sailing. The upgrade to Splunk 8.2.x is fairly minor in comparison with a few great benefits, and then you can upgrade ES over the top of the existing version. There are optimisations available (1) for the KV store, which is highly recommended (the old engine will become obsolete), and (2) for improving indexing with the tsidx writing level, which you can read about once you get to that stage. It's probably the biggest Splunk / ES upgrade you'll need to do for quite a while so triple check your documentation, get plenty of support from your leadership and management, and give yourself plenty of time to perform the change and get it done right. ... View more

When you say multiple notables, do you mean notable events generated when the same correlation search runs again, or notable events generated by a different correlation search? In the first case, you should review the frequency of the alert and the time range it covers. If the alert runs hourly but searches over a 4 hour period, this can potentially alert on the same event 4 times. E.g. if the search condition matches at 12:05, this event would alert at 13:00, 14:00, 15:00 and 16:00 as these 1 hour intervals are all within the 4 hour search period. If you are intentionally including some overlap, you can use a window duration of 4 hours based on fields you want to group by - typically the fields that uniquely define the alert. E.g. if alerting on source and destination IP address, you would group by these fields to prevent the same combination from triggering additional alerts. In the second case, where different correlation searches are triggering on the same events, you likely need to tune your searches. If search A1 is returning event A and search A2 is also returning event A, your searches are likely too broad. Try keeping the search as restrictive as possible, including only the field values you need. If this is not possible, you can filter using where, or using NOT to exclude matching on field values, lookups or subsearches. If you still can't figure out why your searches may be triggering multiple times, please include them here along with their schedule(s), and current windows and grouping fields (if applied). ... View more

It sounds like you're following the correct process, so the best way to test this is to simply generate notable events to confirm your theory. Simple search of "index=_internal | head 1" should suffice. Verify that the notable exists in index=notable and then proceed to the incident review dashboard. If Splunk is otherwise working fine, but you continue to see no new incident review data. review the post installation steps for Splunk Enterprise Security. Try clearing the cache and restarting the browser if required, and restart Splunk if this has not already been performed during the installation process. If problems persist, check the splunkd.log for errors (index_internal source=*splunkd.log sourcetype=splunkd), as well as related components like the mongod service for the KVstore. Details on MongoDB and KVstore troubleshooting can be found at the link below. https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/TroubleshootKVstore If Splunk ES is not performing as intended, there will be logs, and Splunk will provide them. ... View more

If you're referring to the adaptive response in the correlation search itself, as opposed to the contextual dropdown in the additional fields, you could try using a sequence template to pass relevant fields like event_id from the original correlation search into a subsequent correlation search, and then trigger the response from there. Is there a specific reason you want this to be implemented in Python code rather than a GUI-driven workflow? NB: This solution is untested, so please let us know whether this works for you. ... View more

This is the expected result, as the filename is blank in the example provided. | regex field=_raw "FileName.*?\".*?\"(?P<FileName>.*?)\\" To overcome the issue of the trailing backslashes, you can try either escaping the resolved backslash, so the last portion would read as (?P<FileName>.*?)\\\" or (?P<FileName>.*?)\\\\" An alternative is to avoid wildcard matches, such as matching on characters that aren't backslashes. The following should work, but it depends on how Splunk interprets the backslash character in this context: (?P<FileName>[^\]*)" Finally, note that if the filenames themselves contain backslashes, this will break the regex, which is why it's better to remove them at the time of data ingest, allowing you to capture legitimate data without errors, as well as process any escaping that Splunk might (probably will) apply to the file path backslashes. ... View more