The main idea is to have a graceful shutdown/start and run the necessary commands for the cluster’s, had a look at your high-level steps yours looks OK, I did something a while ago, in terms of stopping a Splunk cluster environment and bring them up. By shutting down the data forwarding tier first is a good idea , otherwise the data will be lost nowhere to go. Place the CM in maintenance mode Shutdown Deployment Server / HF ‘s if in use as well Shutdown SHC - take note of the SHC Captain – Stop the SHC Members and Captain Last, make sure they are down. Shutdown Deployer. As the CM should be in maintenance mode  via CM, shutdown shut down the indexers by the way of the normal commands should be fine(/opt/splunk/bin/stop), one at a time and make sure they are down. Shutdown the CM. On the reverse make sure CM  is up and it’s still in maintenance mode, bring all the indexers up and when they are all up  disable the maintenance mode – check status using MC, the replication factors should searchable be green status, so you may have to wait a bit.   Bring back the Deployer back up Then bring the SH's up one by one, ensure the captain is up first, then the other SHC members  and check the others can communicate with it, using SHC clusters commands to check status Bring back the Deployment Server / HF’s Bring back the data forwarding tier. Use the MC to check overall health. I would document all the steps and commands clearly, so you have a process to follow and checkpoint, rather than in an ad-hoc manner due to the many moving parts.    ... View more

As your current inputs is set for scraping all the logs from the folder D:\logs and then you are sending various events from the those logs to null and now you want to be more selective in terms of one log file that you want for info level information and still keep the others from sending some type of events, this becomes a little tricky without testing and having a tinker. Some options that may work: Option 1 You might want to move that log (jkl.txt) to another folder or a sub folder and monitor it separately with another monitor, props and transforms so you can control it, this would leave the other's where they are and you can ingest this one now and filter on it as well. Option 2 Rework your current props and transforms - you may be able to set by source in props, do this for all your other logs and send them to null, either way this all needs some level config and testing out.   [source::...my_otherlog.txt] TRANSFORMS-my_otherlog = my_otherlog_file_null ... View more

Have a look at this one - maybe this would be of help, it mentions Azure App Insights  (It's always worth perusing on Splunk base and working through the Azure TA's as to your requirements). The help file is within the TA so you would need to look at that for further help https://splunkbase.splunk.com/app/7246   ... View more

This is an example using makeresults command - you can use the rex command to extract key values from the  content.payload field  Example only to show you how to extract some of fields I have called my field data replace this with yours . | makeresults | eval data = "fileName=ExchangeRates.csv, periodName=202403, status=SUCCESS, subject=, businessEventMessage=RequestID: 101524, GL Monthly Rates - Validate and upload program" | rex field=data "fileName=(?<fileName>\w+\.\w+),\speriodName=(?<periodName>\w+),\sstatus=(?<status>\w+)" | table * Or look at the spath command  that may also be another way ... View more

I don’t think you can. Setting null routing should come first in your props and transforms (Left to right order) otherwise all the data will get discarded, so look at the order of your props, I'm sure the null is first order which defines the jkl.txt logs.   What you want to do now is to explicitly add the jkl.txt for ingest, so the method would be to whitelist only the files you want to be logged as in the example below.   [monitor://D:\Logs\*] sourcetype = abc index = def whitelist=(*jkl.txt|*myother_files.txt)   So, me thinks you may have to modify the null routing  or disable it. ... View more

Indexed data is set at the bucket (folder) level, Buckets are only frozen deleted or optionally archived when the newest event in the bucket is older than frozenTimePeriodInSecs, therefore you may have a bucket that contains both new and really old data, but the old data won't be frozen until all of the data in the bucket are old (Splunk calculates this in the background) ... View more

I don’t think this is a cert issue, if you use the AOB it tries to validate your app for being certified for the online certificate validation service, basically been given a stamp of approval and needs the below: "Enter the login settings for your Splunk.com account. This information is required for the app precertification process" You normally get this via your sales process.  For the proxy part, it could be incorrect credentials I don’t think it’s a cert issue but could be wrong.   There is a section on the AOB for where self-signed certs should go, but I think this is  red herring https://docs.splunk.com/Documentation/AddonBuilder/4.1.4/UserGuide/ConfigureDataCollection ... View more

First you need to identify what the format is it JSON/syslog/CEF etc is supposed to be. Then you need to install the TA which you have on a HF and in cloud SH.  Then you need to ensure the TA on the HF has been configured with the correct options, you will most likely need to ensure the Guardicore system is configured for the format your require,  or the default options - Speak to the Guardicore admin.  From the Splunk base there appears not to be any detailed documentation but it does state the TA uses REST API and processes events received from the Syslog exporter. So, sounds like the TA app will have the config options to pull data. ... View more

Yes it bit painful,  if you have made lots of /local/ based configs in your apps backup the /opt/splunk/etc/apps folder at minimum, this way you ar least have your app config's backed up and can restore those apps after your re-install Splunk to keep it clean.  ... View more

Have you carefully installed and deployed this add on within your Splunk deployment architecture Follow the instructions https://splunkbase.splunk.com/app/4564 - click on the link and look for where to install this add on section first. You would typically be install this onto a heavy forwarder if you are using one and set the inputs up, this would forward the data to the indexers and data will be parsed. The add is required on the Search Heads for parsing (Knowledge Objects) so needs to be installed there, into the correct path. So Install everythings as required, configure it and then look at the logs. If you have already configured as required then this log message indicates something else. It states "The system cannot find the path specified" Have you installed it correctly? ... View more

Your choices are to work on the F5 LB, speak to your network team for the VIP/pool/failover/irules config and test it out as to work works best.(Its not my area of experteise, I'm just concept aware) Note:The Splunk UF is not a load balancer in the networking sense (It contains Auto Data Loadbalancing function to spray the data across multiple indexers if you have miltiples of them, and to even the data out, its not desinged failover based on load to another UF). The UF is an agent to collect data and send it to Splunk. ... View more

In short there will be a number of installs and configurations you will have to perform, follow this for Splunk Strem components. https://docs.splunk.com/Documentation/StreamApp/8.1.1/DeployStreamApp/AboutSplunkStream High Level Steps 1.Design your architect and the data flow - ensure all ports required can communicate 2. Ensure your device can send netflow data 3. Configure a Splunk index for the netflow data in Splunk 4. Configure Splunk HEC in Splunk and set it to the netflow index 5. Install and Configure the Splunk netflow componenets (ISF) this is where you would point the device to send its netflow data, which then sends to Splunk HEC. Follow this for Splunk HEC configuration https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector Flow this for your architecture - there is a good diagram that you need to consider for all the componenets https://docs.splunk.com/Documentation/StreamApp/8.1.1/DeployStreamApp/DeploymentArchitecture Example of stream in action https://lantern.splunk.com/Data_Descriptors/DNS_data/Installing_and_configuring_Splunk_Stream ... View more

This appears to be a bug - I would log a support call for this. (You may need to uprade)  ... View more

Sounds like you need to to in your custom python script - create a function / code that looks at the inputs file for the key=value as a variable and use that This is a simple test that you can get the value you want - but you will have to dev the code in your python script python3 -c 'print(open("/Splunk/etc/apps/$app/local/inputs.conf").read())' | grep 'value-name = value' ... View more

You say "I have a single server w/ Splunk Enterprise installed on it, as well as SplunkForwarder" Why did you install a forwader onto the Splunk Server instance as well - There is no need to do this, its not a normal thing to do, hence why you are most likley getting those conflict fails.  I suspect you wanted to collect logs etc from the Splunk instance, hence you may have done this, but the full splunk instance will have this functionality built it in. Keep the Splunk instance clean (only installs apps/ta's etc).  Install the forwarder for your target hosts that you want to monitor for your logs etc. If you now uninstall the the forwarder from the Splunk server, you may get all sorts of errors, and then need to re-install the Splunk server as you may have overwritten various Splunk server files etc...messy.      ... View more

This sounds like and LB issue and not Splunk. As to why your F5 is not switiching it might be due to the continuous stream of syslog data being sent, so therefore you will need check your F5 LB conifg options such as round-robin/least connections etc, and ensure its configured for Layer 4 routing and test it out. When using Splunk instances such as HF's as syslog receiver's its generally for testing and non-production enviroments. Why, because if you restart the HF you will loose data for UDP sources,  syslog is Fire and forget and Syslog as a protocol is not ideal for load balancing, so if you can live with the fact you can lose data then so be it. Other issues you can get are, data imbalance on the indexers,data not being parsing correctly as the TA's need reconfiguring to handle sourcetype / parsing when sending syslog to Splunk receiver ports. The best practise for Splunk production enviroments and syslog data are Splunk SC4S and if HA is required then look at KeepaliveD(Layer 4) or Vmotion for HA. SC4S can handle the data and apply metadata for parsing and many other features to effectivly handle common syslog data. LB and HA are two different concepts. ... View more

Here's an example, you can then change to your SPL fields | makeresults | eval millis_sec = 5000 | eval seconds = millis_sec/1000 | table millis_sec, seconds ... View more

From what your saying, something seems then to be overiding it, if its still taking the old setting, which could be another app. Can you show me the output of this command on the UF NOT deployment server? (Obviously remove your hostname and ip for security reasons) /opt/splunkforwarder/bin/splunk btool deploymentclient list --debug Can you also check the log on the UF  (It may help further as to why - should show connection failures at this stage) cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep DC:DeploymentClient Can you confirm the UF can communicate to port 8089 which is the Deployment Server (telnet to it if you can ) temporarly disable the firewall if you can.  Check the Deployment Server ports run the below on the Deployment Server netstat -tuplna ... View more

#On your Forwarders Check this to show what the target is? /opt/splunkforwarder/bin/splunk show deploy-poll On your Forwarders Check this to show what the config is? /opt/splunkforwarder/bin/splunk btool deploymentclient list --debug It might be that the configuration has been set into the below system local config (/opt/splunkforwarder/etc/system/local/deploymentclient.conf) or sometimes its in a custom app (the above Btool should show you this?) If so then change it to the new address (Ensure firewalls and ports are accessable): /opt/splunkforwarder/bin/splunk set deploy-poll <IP_address/hostname>:<management_port> /opt/splunkforwarder/bin/splunk restart   ... View more

Data rolls off due to a few reasons. Data arrives in Splunk, it then needs to move through HOT/WARM>COLD/FROZEN, otherwise data will build up and you will run out of space. 1.Warm buckets move cold when either the homePath or maxWarmDBCount reach their limits.  2.Cold buckets are deleted when either the frozenTimePeriodInSecs or maxTotalDataSizeMB reach their limits. This may help show why it’s moving – see the event_message field index=_internal sourcetype=splunkd component=BucketMover | fields _time, bucket, candidate, component, event_message, from, frozenTimePeriodInSecs, host, idx,latest, log_level, now, reason, splunk_server, to | fieldformat "now"=strftime('now', "%Y/%m/%d %H:%M:%S") | fieldformat "latest"=strftime('latest', "%Y/%m/%d %H:%M:%S") | eval retention_days = frozenTimePeriodInSecs / 86400 | table _time component, bucket, from, to, candidate, event_message, from, frozenTimePeriodInSecs, retention_days, host, idx, now, latest, reason, splunk_server, log_level   You apply config via indexes.conf for the index for disk constrains by configuring the various options: Settings: frozenTimePeriodInSecs (Retention Period in seconds - Old bucket data is deleted (option to freeze it) based on the newest event - maxTotalDataSizeMB = (Limits the overall size of the index - (hot, warm, cold moves frozen) maxVolumeDataSizeMB = (limits the total size of all databases that reside on this volume) maxWarmDBCount = (The maximum number of warm buckets moves to cold) maxHotBuckets = (The number of actively written open buckets - when exceeded it moves to warm state) maxHotSpanSecs = (Specifies how long a bucket remains in the hot/warm state before moving to cold) maxDataSize = (specifies that a hot bucket can reach before splunkd triggers a roll to warm) maxVolumeDataSizeMB = (Overall Volume Size limit) homePath.maxDataSizeMB = (limit the individual index size) coldPath.maxDataSizeMB = (limit the individual index size) maxVolumeDataSizeMB = (limits the total size of all databases that reside on this volume)   See the indexes.conf for details https://docs.splunk.com/Documentation/Splunk/9.2.1/Admin/Indexesconf ... View more

It sounds like you have created a custom syslog app with custom application type of data and its not one of the common NETWORK  syslog sources...this means it’s not going to be parsed and formatted and handled by SC4S, therefore your options are:   Option 1. See if the SC4S community can create one for you (As this sounds like it’s NOT network data then you might have issues as it sounds like a custom application data. SC4S is not designed to handle OS or Application data. You can log an issue here https://github.com/splunk/splunk-connect-for-syslog and maybe they can help. You will need to send a PCAP file. (I doubt if this is feasible, so then look at option 2)    Option 2. Install a normal syslog server (syslog-ng or R-syslog) and configure it as opposed to using SC4S as its primarily designed to handle common network syslog data sources. Send your custom syslog app data to the server running normal (syslog-ng or r-syslog) and configure it log the data into text files into a folder. Install a Splunk UF and configure it to monitor (inputs.conf) your log files and send to Splunk cloud via outputs.conf. The Splunk UF will pick those up and then using outputs.conf send that data to Splunk cloud. You then need to create a TA to parse the custom syslog raw data, so apply metadata, sourcetype, fields, extraction and ensure the timestamp etc are all correct, then install the custom TA in Splunk cloud. ... View more

So the the echo works - you can see data in Splunk, but your syslog APP which sends syslog data is not visable in Splunk and tcpdump shows that the APP is sending data to SC4S. Things to check: 1. Check the “No data in Splunk" section - https://splunk.github.io/splunk-connect-for-syslog/main/troubleshooting/troubleshoot_SC4S_server/#hectoken-connection-errors-aka-no-data-in-splunk Restart sc4s and look at the logs /usr/bin/<podman|docker> logs SC4S 2. Is your syslog APP a common syslog source and supported by SC4S? 3. Is your syslog APP in the known SC4S vendors list? 4. Check if it need some special enviromental config for the /opt/sc4s/env_file (Example look at the McAfee known source, it has a number of configuration options, indexes, ports, TA's env file config see this example - https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/McAfee/epo/) 5. Check the /opt/sc4s/env_file ensure the settings for your syslog APP are set here. 6. Check /opt/sc4s/local/context/splunk_metadata.csv Ensure the keyname (You App source), and ensure its mapped to the correct index in cloud 7. Have you deployed the correct TA's for your syslog APP onto Splunk cloud. ... View more

From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is working. In terms of the /opt/sc4s/local/context/splunk_index.csv follow all the steps from the below run time configuration, there are a number of stpes and you need to complete them all.  https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-runtime-configuration/ As you can send curl test events to cloud you don't need whitelist (BUT its best practise to have them in place for security reasons.)  ... View more