LINE_BREAKER does not extract timestamps.  That's what TIME_FORMAT does. There seems to be two issues here: Lines not breaking correctly when the date is a single digit. Timestamps not extracted correctly when the date is a single digit. There's a separate setting for each. The line breaker is telling Splunk the next event must have a two-digit year so that is what Splunk does.  To make the date 1 or 2 digits, modify the regex: ([\r\n])\[\w{3}\s\w{3}\s\d{1,2}\s\d{2}:\d{2}: or keep it simple with ([\r\n])\[\w{3}\s\w{3} The latter regex has the advantage of handling any time zone. The %d formatting variable accepts both one- and two-digit dates.  I believe the problem with the formatting string is the extra space after "%d". Here's the final set of props.   [sql:logs] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n])\[\w{3}\s\w{3}\s\d{1,2}\s\d{2}:\d{2}:\d{2}\s\w{3}\s\d{4}\] TIME_PREFIX=\{ TIME_FORMAT=%a %b %d %H:%M:%S %Z %Y       ... View more

We can't help you fix what's wrong without knowing what is wrong.  "Red health" could be caused by any of many things (or multiple things).  Click on the read health icon to show the health status and then click on each red icon to get details about what is wrong (or what Splunk thinks is wrong - sometimes it's wrong about that).  Pass on that information and someone should be able to help you fix it. ... View more

Just about every Splunk query contains a pipe so I question the accuracy of the post you quoted. My theory (and I can't find any documentation to support it) is the collect command does not return the field being throttled so the alert cannot be throttled.  The collect command is used to write results to an index and is used to populate summaries more than it is to trigger alerts.  I suggest having two searches - one to populate testindex and other other to read testindex and trigger the alert. ... View more

The first error is in using an indexer as a UDP receiver.  That likely will result in data loss.  Recommended practice for the last several years is to send syslog events to a dedicated syslog receiver, such as syslog-ng or rsyslog.  Then use a Universal Forwarder to send the events from the syslog server to Splunk. The current LINE_BREAKER setting says an event doesn't end until it finds a newline so the incoming text is held until that condition is met.  This can be overridden in file inputs, but not with UDP (another case for using a syslog server).  Try this line breaker and also add TIME_FORMAT. LINE_BREAKER = ([\r\n]+).*$ TIME_FORMAT = %b %d %H:%M:%S.%3N   ... View more

Sometimes you can right-drag and then opt to copy without formatting.  If not, copy the field name and use Ctrl-Shift-V to paste it without the HTML. ... View more

It's not clear what is meant by "partial search" and how Splunk is to know a search returned partial results or just fewer results. The subsearch idea likely won't work because subsearches execute before the main search and so would be unable to detect errors in the main search. There is the require command that will abort a query if there are zero results.  That may not meet your requirements, however. ... View more

From Splunk's perspective, no action is needed to stop using the software.  Your company, however, may have its own requirements, such as archiving the data before decommissioning the server. ... View more

Splunkbase automatically archives apps that have not been updated in 24 months.  That does not mean the app no longer works or cannot be used.  However, as noted on the cited page, this app has been replaced by another. The functionality of this add-on has been incorporated into the supported Splunk Add-on for Microsoft Office 365 https://splunkbase.splunk.com/app/4055 ... View more

It depends on what is meant by "decommissioning Splunk".  There is a process for removing a member from a search head or indexer cluster.  Decommissioning an independent indexer typically would mean moving that indexer's data to another indexer. If you will not be running Splunk anywhere then just stop Splunk and retire the server. ... View more

What is your role?  The indexes you can see are based on your role. Can you share the exact error message(s) so are seeing along with the query that caused?  That will help us find the source of the problem. ... View more

Remove "authorisationheader" from the URL.  That's not a valid HEC URL. If that doesn't help, then please post the exact text of the error message(s) you see and also identify the "it" that tells you the URL is incorrect. ... View more

All tokens in a search must have a non-null value before the search will run. Try setting the null tokens to empty strings ("") before the search. ... View more

A local account is one that does not use SAML or LDAP for authentication.  It's the default if you have not configured SSO. The account you log onto the Splunk console with may not be available on all instances.  The account is know to the search heads, but not to indexers or universal forwarders and probably not to heavy forwarders.  Each could have had an admin account created when Splunk was installed.  If it was not created or if you don't have the password, reset the account using the instructions at https://community.splunk.com/t5/Security/How-to-Reset-the-Admin-password/m-p/10622 ... View more

When a CLI command asks for credentials, it expects a Splunk local account name with admin privileges.   If you do not have the admin password then you should reset it. Except for indexers and universal forwarders, just about any Splunk instance may be using KVStore.  It's also possible a new app will be installed that uses KVStore so it should be running. ... View more

Have you tried removing '/bin' from JAVA_HOME and the config file? ... View more

Splunk does not delete individual events - it removes entire buckets when either the size or time limit is reached. When deleting by time, because the whole bucket is deleted, it's important that all of the events in that bucket be old enough to delete.  If any event is too new then the bucket will not be touched.  Every bucket has two dates (for our purposes, anyway) associated with it - the start date (_time of the first event added) and the end date (_time of the last event added).  The end date is one that determines when the bucket can be deleted/frozen. I've seen sites where data is poorly onboarded and has _time values in the future - sometimes by years.  When that happens, the bucket will remain in the system until frozenTimePeriodInSecs after that future date passes. ... View more

This is fairly common and easy to correct. Stop Splunk Change the ownership of all Splunk files using the command (as root) chown -r splunk:splunk /opt/splunk Start Splunk ... View more

Did you restart Splunk after changing $JAVA_HOME? The info is stored in $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/dbx_settings.conf,  See https://docs.splunk.com/Documentation/DBX/3.18.0/DeployDBX/settingsconfspec for a description of the contents. ... View more

One problem with transferring $SPLUNK_HOME from one Splunk instance to a newer one is you will be taking the old Splunk built-in apps with you, which would not be a Good Thing. Another potential problem is you will miss out on the migration actions taken during upgrades.  If you've upgraded Splunk before, have a look at $SPLUNK_HOME/var/log/splunk/migration.log.* to see what is done behind the scenes during an upgrade.  Without that work, you may be carrying useless (or even harmful) cruft to the new version and may miss out on important changes. ... View more

@jessieb_83 wrote: I understand that maxTotalDataSizeMB takes precedence over frozenTimePeriodInSecs. You understand incorrectly.  Both have equal precedence.  The first limit reached is the one that will be applied.  If the size limit is reached then the oldest buckets will be deleted first. If maxTotalDataSizeMB is not specified, it defaults to 500000 (see indexes.conf.spec). Use the dbinspect command to examine your buckets.  Make sure the oldest ones don't have an earliest_time latest_time (endEpoch) that is newer than the frozenTimePeriodInSecs setting.  Buckets will not age out until *all* of the events in the bucket are old enough. ... View more

Change the Java home setting in DB Connect and/or correct the $JAVA_HOME environment variable. ... View more

The gzip'd files are index under their own source names.  They come in the query because their names match the pattern source="/var/log/nginx/access.log*".  Remove the asterisk and only the one file will appear. ... View more