Have you configured and enabled an input as described in https://splunk.github.io/splunk-add-on-for-amazon-web-services/CloudTrail/ ? P.S. It appears the version 7.10.0 is no longer available to consider upgrading to 7.11.0 or newer. ... View more

We need more information. Are you saying to want to send data collected by Splunk to GCP?  Is that all data or just some of it? Can you be more specific about the destination?  Saying "GCP" is like saying "AWS".  Both offer a large number of services with different requirements and capabilities. What is your SIEM? It may be necessary to write code that uses the API to fetch data and then ship it to GCP. ... View more

Does your Splunk Cloud stack have REST API enabled (it's off by default)?  If not, open a support case. Is the IP address of the server running the script on your stack's IP Allow List?  If not, get a Splunk admin to add it. ... View more

They are not "steps".  They're separate checks/measures to perform to try to alleviate delayed searches. See https://help.splunk.com/en/splunk-enterprise/get-started/deployment-capacity-manual/9.4/performance-reference/reference-hardware and https://help.splunk.com/en/splunk-enterprise-security-8/install/8.1/planning/minimum-specifications-for-a-production-deployment Enterprise Security is a very resource-intensive application.  Therefore, it is recommended to install ES on a separate Splunk instance.  It can, however, share indexers with other search heads. This should require no explanation. Real-time searches pin themselves to a CPU, preventing other searches from running there.  Don't use real-time searches.  See https://help.splunk.com/en/splunk-cloud-platform/search/search-manual/9.2.2406/search-and-report-in-real-time/about-real-time-searches-and-reports for more. Entire books could be written on making searches more efficient.  Splunk has one at https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.4/optimizing-searches/about-search-optimization Allow Skew gives the search scheduler permission to adjust the run time of a scheduled search to one with fewer other searches scheduled.  Search Windows allow the scheduler to delay the start of a scheduled search in the event that resources are not yet available.  See https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/reporting-manual/10.0/report-management/offset-scheduled-search-start-times and https://www.splunk.com/en_us/blog/platform/schedule-windows-vs-skewing.html There is a strong tendency among Splunk users to run their scheduled searches at the top of an hour.  At most of the customers I've visited, this accounts for about half of all scheduled searches and is a source of most of their delayed and skipped searches.  It also doesn't account for the 30-90 seconds of delay between when an event is generated and when it is searchable by Splunk.  It's far better to use a cron schedule to have the search run at 2-3 minutes after the hour.  Other peak search periods to avoid are 15, 30, and 45 minutes into any hour of the day. Splunk's Workload Management feature gives Splunk admins some control over how resource contention (CPU and memory) is handled.  It also can be used to stop long-running searches, prevent real-time searches, and prevent users from running searches during peak times.  See https://help.splunk.com/en/splunk-enterprise/administer/manage-workloads/9.4/workload-management-overview/about-workload-management for details. I've seen plenty of instances where a reports run once a day or even every week at 8 or 9 in the morning.  This usually is unnecessary and takes away "slots" from other searches.  Instead, these types of "batch" reports should run in less busy times of day such as 3am or on weekends.   ... View more

If there were no events in the last 24 hours then there is no last timestamp to display. The only way to get the timestamp would be to join the current search with one that scans the logs for the most recent entry for each sourcetype in some larger time window (perhaps 30 days). ... View more

It's not clear if you want to correlate Splunk data in your ITSM product or ITSM data in Splunk. There are a few ways to onboard data into Splunk. Install a universal forwarder on the server to send log files to Splunk Have the server send syslog data to Splunk via a syslog server or Splunk Connect for Syslog Use the server's API to extract data for indexing Use Splunk DB Connect to pull data from the server's SQL database. Have the application send data directly to Splunk using HTTP Event Collector (HEC). There are REST API calls available for getting data out of Splunk. Let us know which you are trying to do and we'll point you toward to relevant docs.   ... View more

Use the table command to put the fields in the desired order | chart sparkline count by a,b | table a b count sparkline   ... View more

There is no way to mitigate the vulnerability except to install the next version of Splunk that includes a fix.  We don't know which version that will be, but Splunk typically announces when vulnerabilities are fixed.  Check out https://advisory.splunk.com/ ... View more

Thanks for clarifying. While Splunk Enterprise can be installed on Windows workstations, that's typically done for dev/test purposes.  Workstations don't have the horsepower to handle enterprise quantities of data. The typical model is to install SE on shared servers (on-prem or virtual) with all other enterprise devices running a Universal Forwarder to send data to SE. ... View more

These are two different problems. For the first, know that Splunk indexes are immutable.   Data cannot be removed selectively.  My advice is to copy  the desired data to a new index (use the collect command) and then delete the contents of main. For the second, I presume you're referring to the list of forwarders in the Monitoring Console.  The "missing" forwarders can be removed by clicking the "Rebuild asset list" button. ... View more

Why are you installing Splunk on every Windows system?  It should be installed on a central server and accessed via web browser. Perhaps you're referring to the Splunk Universal Forwarder.  That should indeed be installed on each machine, but do it as part of the system image.  If that's not possible then I know customers have installed it using GPO. To prevent Splunk from asking for credentials, supply them when launching the installation.  See the installation manual.   ... View more

You may want to contact the developer directly at [email protected] ... View more

Searches are delayed because too many searches are trying to run at the same time.  There are a few things you can do about it. Ensure all instances meet or exceed Splunk's Reference Hardware specification. Ensure Splunk Enterprise Security runs on a dedicated Search Head (or SHC).  Do not run searches unrelated to ES on that SH. Eliminate unneeded scheduled searches. Prohibit real-time searches. Verify all scheduled searches complete as quickly as possible.  Do this by minimizing their search windows and using efficient searches. Implement Allow Skew and Schedule Windows. Distribute search run times evenly across the clock.  Avoid running at peak times such as 0, 15, 30, or 45 minutes of each hour. Consider using Workload Management to control the search behavior of users. Move daily/weekly/monthly scheduled searches to off hours. ... View more

Please contact your Splunk account team or the Partner from which you purchased Splunk for answers to pricing questions.  Only they will have the correct answers. ... View more

No, you are not losing "Running" from each event.  Only the first capture group in LINE_BREAKER is discarded from the event. However, positive and negative lookaheads do not perform well in Splunk so they should be avoided. ... View more

Always.  It never hurts to have it and can help when the UF has to break an S2S packet.  Don't forget to also set EVENT_BREAKER_ENABLE=true. ... View more

Positive lookahead wastes resources.  Over millions of events, those extra cycles add up to excess SVC consumption.  This works better LINE_BREAKER = ([\r\n]+)Running ... View more

It's not that simple and we don't have enough information.  Have a conversation with your Splunk account team and they will be able to give you the answer that fits your environment. ... View more

Subsearches are performed first, before the main search executes.  That's why there's nothing for the lookup command to find. ... View more

What you're trying to do seems like overkill for a data dictionary.  It also ignores queries contained in saved searches.  I don't see how parsing search queries helps build a data dictionary; although I can see how it might show what data people are searching for. When I work with my customers to build data dictionaries, we use nightly scheduled searches to pull lists of sources and sourcetypes from the indexes.  Using the tstats command makes this fast and lightweight. | tstats count where index=* by index,sourcetype | outputlookup DD_sourcetypes.csv.gz Once the lookup file is created, you can use it to build your Data Dictionary dashboard. Don't forget to add logic that trims the lookup file or else it might grow forever. ... View more

1. This information is available from the eai:acl.app and label fields, respectively. 2. Not all panels have a name.  To get the names, parse the eai:data field and pull out the dashboard.row.panel.title and form.row.panel.title elements. | rest splunk_server=local /servicesNS/-/-/data/ui/views | fields eai:acl.app label eai:data | rename eai:* as * | spath input=data | rename dashboard.* as *, form.* as * | fields acl.app label row.panel.title Note that this SPL works only with Simple XML dashboards.  That's because Dashboard Studio uses a very different name scheme for elements and because the spath command fails on DS dashboards because it contains JSON nested within XML. 3. This is tricky because base searches can exist both within and without a panel.  Also, you'll probably want to retrieve the earliest and latest elements for each query. 4. This is in the row.panel.search.query and row.panel.*.search.query elements, however, not all panels have a query. 5. This is another tricky bit.  The data returned above will be multi-valued since there may be many elements with the same name ("row.panel.table.search.query", for example).  Mapping a panel name to its query will be a challenge. If you still want to attempt this (and, like @ITWhisperer , I wonder about the utility of this), then consider using Python instead of SPL to do it. Depending on the use case, it may be possible to get much of the same information from Splunk's internal logs.  Tell us more about the use case so we might offer other solutions. ... View more

Have you tried fixing the inputs.conf error to see if that makes a difference? ... View more

It may help to enumerate the data fields rather than have Splunk extract them from the input file.  Use the FIELD_NAMES setting to do that.  See https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configuration-file-reference/10.0.2-configuration-file-reference/props.conf#:~:text=set)%3A%20not%20set-,FIELD_NAMES,-%3D%20%5B%20%3Cstring%3E%2C...%2C%20%3Cstring%3E%5D%0A*%20Some ... View more

Contact your Splunk account team.  They will look at your Splunk usage and will be able to determine if you can save any money by switching to workload pricing. ... View more