Yes, I suspected that would happen, maybe try: 1. Stop Splunk if you can 2. Backup /opt/splunk/etc/apps folder  (So you have your App configs at least) 3. For your data if you are using the default in $SPLUNK_HOME/var/lib/splunk folder - you can be move to a temp folder as well, but if you had a seperate volume even better - it wont get touched. 4. Re-install Splunk over the current broken install  and see if that works (I suspect not) but worth a go 5. If it works restore the /opt/splunk/etc/apps folder and your data . (make sure you set the splunk permissions - chown -R splunk:splunk etc to the Splunk and data folders If that all fails, then may be wipe it clean and start again, if you keep it as it is not going to bode well for the future as you will have other upgrades to do in the future  and it will always cause some kind of problem, so better to sort it all out now and make it clean. If it was me, I would start clean again, less issues in the longer run. ... View more

Find your Knowledge object and the owner - look at the below example and change as to your requirments. example curl -k -u admin_user:password https://<MY_CLOUD_STACK>splunkcloud.com:8089/servicesNS/nobody/YOU_APP/saved/searches/my_search/acl -d 'owner=new_user'-d 'sharing=global' -X POST Heres some further help on ACL in cloud https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/RESTTUT/RESTbasicexamples ... View more

Sizing Splunk for ES use has many factors for performance and to consider, it’s not a one size fits all. As everything is a search, you need to have sufficient resources to cater for users and various aspects of the Splunk environment alongside the ES functions. We go by a rule of thumb for ES sizing 100GB Per indexer, I have seen this higher in some cases (so the amount you’re ingesting per day), so try to understand much volume of data ingest comes into you Splunk per day. We typically dedicate the ES SH on its own for large environments, the reason is when data comes in Splunk it will also be placing that data to disk, being a provider of that data, searching the data, running datamodel searches for the correlations rules and there will be dashboards. On top of this you may have other users using the ES or ad-hoc searches - so you can see many aspects to consider (CPU/RAM/IO/Network), otherwise it can become slow and you don’t want that, you need results in a timely fashion. As guide its best for minimum for 16CPU/32G RAM for indexers and SH - as you have 32CPU/32RAM you should be ok as a starting point,  but that does depend on the workload. You also need to check that the disk is SSD and I/Ops is over 800 and ensure you are not sending large volumes of data per day so that your AIO AIO can’t handle all the functions - so keep a check on ingest per day. How to check for Correlation Searches resources consumption? I would start to use the monitoring console(MC) for the usage stats, it’s very comprehensive, it will show the load etc, you can see which searches are consuming memory and this will help you with some aspects of resources.the MC comes with Splunk so it should be on your AIO. - see my links below for refence. Some tips: Ensure you only ingest important data sources on boarded and they are CIM Complaint via the TA's Enable a few data models at time based on your use cases (Correlation rules you want to use) and keep monitoring via the MC checking the load overtime, this will help you keep on top of the resources. Here ‘s some further links on the topics I have mentioned that you should read. ES Performance Reference https://docs.splunk.com/Documentation/ES/7.3.1/Install/DeploymentPlanning MC Reference https://docs.splunk.com/Documentation/Splunk/9.2.1/DMC/DMCoverview Hardware Ref https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware ... View more

 Yes, I understand it’s the copy part, you want, unless the system logs it its going to be tricky.  This may be of help not sure, so now I remember many years ago, I helped a company, to monitor for when their important/sensitive files in a Windows Server were being accessed/copied or deleted.   We got the Windows Administrator to enable security/auditing for the important files etc (read/write/delete attributes) etc.  We then tested for files being copied/deleted/accessed, the event ID's were something like below, and we sent these to Splunk and was able to monitor them for insider threats. (This was only done for the most important files as they could generate lots of events)  If you enable these they may contain more info as to the device as well, cant remember, but worth a go.  Event ID 4656: A handle to an object was requested. Event ID 4663: An attempt was made to access an object. Event ID 4660: An object was deleted. ... View more

Maybe you need to have these enabled, work with your Windows Admin to get them enabled or check for you. Or simple insert a drives and look for the logs. Here are some common event IDs that I was able to find using Google! Some may be logged others not when drives are added etc, but you will have to work with your Windows Admin to find out if they have been enabled or are logging etc or use sysmon which is a tool to also monitor and create events. I have listed the TA's that you will need to use once you have all the logs in place. Event ID 12: This event is logged when a removable media is inserted into the computer. Event ID 106: This event is logged when a new external storage device is connected to the system. Event ID 20001: This event is logged by Microsoft-Windows-DriverFrameworks-UserMode when a new device is connected to the system. Event ID 20003: This event is logged by Microsoft-Windows-DriverFrameworks-UserMode when a device is removed from the system. Event ID 1001: This event is logged when a device is enumerated by the Plug and Play manager. Event ID 1003: This event is logged when a device is started. Event ID 1004: This event is logged when a device is stopped Splunk TA's Windows Logs https://splunkbase.splunk.com/app/742 Sysmon logs / info https://splunkbase.splunk.com/app/5709 ... View more

Splunk will report on the data it has, so you have to first identify which logs or other data sources contain the data that shows data being copied to a removable media - so this is something you have to find out based on your systems. Once you have this knowledge of where the data is you will need to ingest the data source into Splunk, extract fields and use them to report via Splunk. ... View more

Hi Follow all the steps and plan your upgrade from the below link, there many steps and processes to follow. Most important always Read the README for the version you upgrading. https://docs.splunk.com/Documentation/Splunk/9.0.1/Installation/HowtoupgradeSplunk ... View more

I dont know about the exact postman config for filtering, but via CLI you can test the below first and assuming you can use a Linux syste. . For the API call its seems to be called name and not title as I have noticed, this is difference between | rest and calling the API. (dont know why this is...) Further more If you install the jq command it’s a json processer command, it will help with the two fields you want, if not remove from my command below. You will need a token created in Splunk. See my example below curl -k -H "Authorization: Bearer <YOUR TOKEN>" https://*****:8089/servicesNS/-/-/admin/macros --get -d output_mode=json | jq '.entry[] | {name: .name, definition: .content.definition}' This should give you the results for the name of the macro and its defintion, optionally output to a json file ... View more

The forwarders (UF) or HF have inbuilt functionality to send the data to both, so as long as you configure in the outputs.conf, the group names of the servers. See the section "Configure load balancing on a universal forwarder with outputs.conf" https://docs.splunk.com/Documentation/Forwarder/9.2.1/Forwarder/Configureforwardingwithoutputs.conf See this document for how autoload balancing works https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Forwarding/Setuploadbalancingd ... View more

This is the command if using Linux - /opt/splunk/bin/splunk resync shcluster-replicated-config (Run this on other SHC members – not captain)   Check the status first, runs on one of the SHC members /opt/splunk/bin/splunk show shcluster-status But might be worth trying a rolling restart and see if that helps, looks like the /var/run folder on the captain is having some kind of issue. Check Disk Space as well. du -sh /opt/splunk/var/run to get the size of the folder. Rolling restart command = /opt/splunk/bin/splunk rolling-restart shcluster-members from one of the members Monitor the rolling restart /opt/splunk/bin/splunk rolling-restart shcluster-members  -status 1 The captain may change – so observe – you can change the captain back to original /opt/splunk/bin/splunk transfer shcluster-captain -mgmt_uri <your SHC Captain> If this is production, then factor in maintenance window or do it when least busy for users, as it will be somewhat disruptive for searches, until it's resolved. But it It's worth going through the previous answers for this issue, the commands and various steps. Ensure you have backups and plan the steps for the procedure. At minimum /opt/splunk/etc which contains the configuration. https://community.splunk.com/t5/Deployment-Architecture/How-to-resolve-error-quot-Error-pulling-configurations-from-the/m-p/354231 https://community.splunk.com/t5/Deployment-Architecture/How-do-I-fix-quot-splunk-resync-shcluster-replicated-config-quot/m-p/212358 ... View more

At a high level: You need to find a method of the Jmeter application to either dump/send/ftp/scripts  etc the csv files into a folder and create an inputs.conf with the required config(props.conf) file to monitor the csv files. Where the csv files will be located you will need a UF configured so that it sends to the Indexers. The csv files should contain a time stamp, this should comes from the Jmeter application, and hopefully it does,  otherwise it will use default mod time. In the inputs.conf there's an option to monitor the files and then delete them, this is the batch option so once it monitors the csv file, it sends it to the sinkhole (deletes it), other wise use the normal monitor option, your choice. The it’s the goal is achievable, needs some setup/config/testing/refining/data checking. ... View more

You can check via the below REST command in the Splunk search bar on the Deployment Server to see the settings  | rest splunk_server=local /services/configs/conf-serverclass | fields crossServerChecksum, disabled, endpoint, filterType, id, repositoryLocation, restartSplunkWeb, restartSplunkd, splunk_server, stateOnClient, targetRepositoryLocation, title, whitelist.0 | rename title AS server_class_name | table server_class_name, whitelist.0, crossServerChecksum, disabled, endpoint, filterType, id, repositoryLocation, restartSplunkWeb, restartSplunkd, splunk_server, stateOnClient, targetRepositoryLocation ... View more

It looks like, you have the data, why not go through some training on creating some generic stuff, this is free online guide, good starting point, you learn some basics concepts, you can apply the principles to the AD data, and then further develop your skills by looking formal training ones.  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/SearchTutorial/WelcometotheSearchTutorial ... View more

What I would say is we would normally send all the data to both indexers (This is data load balancing , portions of the data get spread across the two indexers and you get better performance that way, this also has nothing to do with data clustering in the real sense - Best preactise. But it’s your choice at the end of the day. Heavy Forwarders are typically used for Add-ons – full Splunk instance, from there they send data to indexers. Best practice. You should also send the HF internal logs to the indexers – Best Practise. The SH connects to the indexers and is configured to sends its logs to the indexers, Best Practise. The below is an outputs.conf example  that sends to both indexers and sends the local Splunk internal logs , you can tune it to just send to one indexer if you want, just remove the second indexer from the group list and uncomment the specific indexer setting. Add this to your $SPLUNK_HOME/etc/system/local/outputs.conf on the SH - make the changes to reflect your environment names and test. If you’re using a custom separate app for outputs.conf then add to that. Restart the SH Splunk. You can do the same on a HF. NOTE:(The new internal indexes _ds*, so these need to be created on the indexer's if you are using the latest versions of Splunk) . NOTE: Ensure firewalls, ports, NTP have been configured and I'm assuming your not using TLS - thats another subject.    outputs.conf  Example [indexAndForward] index = false [tcpout] defaultGroup = <my_group_name_indexers> forwardedindex.filter.disable = true indexAndForward = false forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent) [tcpout:my_group_name_indexers] #Remove the second indexer if you only want to send to one indexer server = mysplunk_indexer1:9997, mysplunk_indexer2:9997 #This is only for one indexer receiver #[tcpout-server://mysplunk_indexer1:9997]   ... View more

Without giveout more information so we can help you (it's better to provide more context as to your issue, screen shots etc) that said it sounds like its related to risky commands. Maybe its to do with this. https://docs.splunk.com/Documentation/Splunk/9.2.1/Security/SPLsafeguards ... View more

You should not be instaling indexes on the SH - this is just for search purposes - data is stored on on the indexers and your indexes to should be set there . From your comment "so the collected logs are stored in the search head's index"... this is not the way The App/Add-on contain knowledge objects which are used for things like dashboards and parsing search time data, the TA only should be installed onto the indexer or Heavy forward if that is where the data is sent to first. As to your comment "I set up a new instance for the heavy forwarder on which I install the add-on, and I configure it to forward the indexes to the Indexer."  This is the way forward.  ... View more

To get you started here's a number of links for you read and work through In short you need the nix TA, UF and configure inputs and outputs based on your requirements. #This shows you the TA Required (Nix TA) https://splunkbase.splunk.com/app/833 #This shows you the OS Supported = MacOs is listed https://docs.splunk.com/Documentation/AddOns/latest/UnixLinux/About   #Read the release notes https://docs.splunk.com/Documentation/AddOns/released/UnixLinux/Releasenotes And you will need to install a Universal Forwarder for the MacOS + configure outputs and TA inputs https://www.splunk.com/en_us/download/universal-forwarder.html ... View more

It could be a n umber of things as to why the data is not coming through or not showing. 1.Whatever your monitoring does it have read permissions? 2.Check for typos' (index name etc) You can also check the internal logs for clues index=_internal sourcetype=splunkd host=neo log_level=INFO component=WatchedFile | table host, _time, component, event_message, log_level | sort - _time   What is the output of this command - it shows whats being monitored (Assuming its a linux host) /opt/splunk/bin/splunk list inputstatus   Are you able to show us your inputs.conf and describe what you are trying to monitor? ... View more

 Network data can be notorious for sending large volumes of data - where possible filter at source.   It’s also worth thing about how you’re sending the network data to Splunk   The  better syslog options are: Splunks free SC4S (container syslog under the hood) Have a syslog (r-syslog or syslog-ng) server and send the data there, then let a UF to pick up from there, and send it to Splunk.   Many people set up TCP/UDP ports on a HF or Splunk Indexers, and this can various implications for large environments (not saying you can't do this) but it’s not ideal for production, but for testing or small environments Ok. ... View more

Splunk is very security focused and has many apps and various options for data etc.... A couple of tips: When I first started, I had a play with this App, its a great way to learn about security datamodels and dashboards. Its not Splunk ES SIEM but its still very good to have a play with and learn. https://splunkbase.splunk.com/app/4240 The other one to install and have a play with is security essentials - this provide so much security content, again good to learn about. (This is not a SIEM or monitoring app)  and its my go to for security use cases. https://splunkbase.splunk.com/app/3435 ... View more

  There are many factors that could cause performance issues in your prod environment that wasn’t in the dev environment, production normally has more data and many other variables to that could cause issues.   Splunk is a workhorse, it needs CPU/Memory/Disk resources and other factors to be in place. Things to consider Has the environment been sized according for production? Is the disk on fast type of disks SSD etc Are  there lots of user’s run running the same search and for all time and at the same time? Do you have indexer clustering or is it Splunk All in One The Add-ons (TA’s) normally provide parsing and other knowledge objects  and potentially it could impact the environment with regex processing as an example.  The Splunk apps on the other hand have searches and dashboards that could potentially impact with long running searches. But normally it’s down to the Splunk sizing or something based on the environment. I don’t re-call a TA ever causing performance issues in the PROD environment, but it could happen I guess. I suggest: Use the  Monitoring Console for the production environment, this is a good place to start to check the performance issues. Check CPU/Memory on the SH and Indexers first. Check the Searches Run and Search Memory usage,  using the  MC. If you removed the TA does it improve and re-install, does it get bad again? If that all fails then perhaps look at logging a support call.   Monitoring Console https://docs.splunk.com/Documentation/Splunk/9.2.1/DMC/DMCoverview Splunk Sizing guide https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Sizing_your_Splunk_architecture ... View more

From the MC run the below - it should give you a starting point index=_internal source=*metrics.log group=tcpin_connections fwdType=uf hostname=* | eval hostname=lower(hostname) | fields _time hostname sourceIp arch destPort fwdType os ssl version | table _time hostname sourceIp arch destPort fwdType os ssl version | dedup hostname ... View more

The main idea is to have a graceful shutdown/start and run the necessary commands for the cluster’s, had a look at your high-level steps yours looks OK, I did something a while ago, in terms of stopping a Splunk cluster environment and bring them up. By shutting down the data forwarding tier first is a good idea , otherwise the data will be lost nowhere to go. Place the CM in maintenance mode Shutdown Deployment Server / HF ‘s if in use as well Shutdown SHC - take note of the SHC Captain – Stop the SHC Members and Captain Last, make sure they are down. Shutdown Deployer. As the CM should be in maintenance mode  via CM, shutdown shut down the indexers by the way of the normal commands should be fine(/opt/splunk/bin/stop), one at a time and make sure they are down. Shutdown the CM. On the reverse make sure CM  is up and it’s still in maintenance mode, bring all the indexers up and when they are all up  disable the maintenance mode – check status using MC, the replication factors should searchable be green status, so you may have to wait a bit.   Bring back the Deployer back up Then bring the SH's up one by one, ensure the captain is up first, then the other SHC members  and check the others can communicate with it, using SHC clusters commands to check status Bring back the Deployment Server / HF’s Bring back the data forwarding tier. Use the MC to check overall health. I would document all the steps and commands clearly, so you have a process to follow and checkpoint, rather than in an ad-hoc manner due to the many moving parts.    ... View more

It appears the the guardduty logs are collected via cloudwatch which this TA supports (https://splunkbase.splunk.com/app/1876), so this is most likely what you need. I think the old TA's used ot be seperate and now they have been combined into this one TA. See the different sourectypes - for you its aws:cloudwatchlogs:guardduty https://docs.splunk.com/Documentation/AddOns/released/AWS/DataTypes General info on thie TA https://docs.splunk.com/Documentation/AddOns/released/AWS/Description ... View more