DATETIME_CONFIG is usually not changed across sourcetypes (it's very rarely touched) except for setting it to CURRENT or - much less often - NONE. It's the other settings which are responsible for timestamp extractions. Key settings here are /opt/splunk/etc/apps/TA-linux_auditd/default/props.conf TIME_FORMAT = %s.%3N /opt/splunk/etc/apps/TA-linux_auditd/default/props.conf TIME_PREFIX = msg=audit\( That's what I was talking about all along. If Splunk cannot find a timestamp for the event, it assumes current (in this case - midnight) event. Then, until it hits limits for MAX_DAYS_AGO, MAX_DAYS_HENCE, MAX_DIFF_SECS_AGO and MAX_DIFF_SECS_HENCE, it just assumes every subsequent event has the same timestamp since it cannot find a "healthy" timestamp in the event itself. ... View more

@splunkisaurus  Why dont you output JSON from your script so Splunk ingests clean structured events. You can also use props.conf if you need to split the events. In that case, you can rely on LINE_BREAKER alone and omit both SHOULD_LINEMERGE and BREAK_ONLY_BEFORE For eg: [nessus_agent_status] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE = ^Running: LINE_BREAKER = ([\r\n]+)Running: TRUNCATE = 0 DATETIME_CONFIG = CURRENT Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

Hi @richgalloway @livehybrid @PrewinThomas .. all three of you gave the right answers. so, following the "first come first served" logic, selecting Rich's answer as the solution. given karma points for all 3, thanks.  for future users reference, pls check these 2 links for detailed info: https://help.splunk.com/en/data-management/splunk-enterprise-admin-manual/9.4/administer-the-app-key-value-store/back-up-and-restore-kv-store https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/administer-splunk-enterprise-with-the-command-line-interface-cli/administrative-cli-commands   ... View more

Hi @kamalKSharma  there are only these 3 ideas: 1) there is a "Splunk Enterprise Security Guided Product Tour" https://www.splunk.com/en_us/form/enterprise-security-tour.html 2) on youtube, there are few good Splunk ES demo videos available(they are from Splunk's own channel, so very good quality materials they are.) 3) as said by @PrewinThomas , splunkbase restricts "trial" downloads to customers who have purchased ES or have been provisioned a trial entitlement(by Splunk Sales team or Splunk Account mgr) ... View more

@trevorharris  You don’t necessarily need to disable the FortiGate App before upgrading Splunk to v10. The app is officially listed as compatible only up to Splunk v9.4, which means vendor-tested support stops there. However, in practice, many Splunkbase apps continue to function on newer releases even if compatibility hasn’t yet been updated. So better test it on a test/dev instance(splunk 10) to make sure everything works as expected. Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

Hi There Dear Splunkers,  if you are thinking to check the Splunk docs for this task, here it is: https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.4/deploy-search-head-clustering/integrate-the-search-head-cluster-with-an-indexer-cluster   if my post helped you in anyway, a karma would be helpful, thanks.  ... View more

Awesome, that did the trick, thank you! For those reading this in the future, running @PrewinThomas command from the Splunk bin reloads the DB Connect settings without having to restart Splunk/SplunkD/the splunk server. ... View more

@rayleigh29  Did DB Connect function previously? Please verify file permissions for the DB Connect app and ensure the user has adequate read/execute access. splunk stop chown -R splunk:splunk $SPLUNK_HOME/etc/apps/Splunk_DB_Connect chmod -R 755 $SPLUNK_HOME/etc/apps/Splunk_DB_Connect If your kvstore is corrupted, clean the kvstore $SPLUNK_HOME/bin/splunk clean kvstore --app Splunk_DB_Connect Then start splunk and test again. Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

@ThuLe  Most probably network related. -Check connectivity to Splunk Cloud - telnet inputs1.STACKID.splunkcloud.com 9997 -Check Firewall/SSL inspection recently enabled or changed #https://splunk.my.site.com/customer/s/article/Splunk-Universal-Forwarder-is-not-sending-events-to-Splunk-Cloud Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

It's the main application that in some cases gets crazy and start writing so many error headers ☹️ i still talk to developers and tell this is the why, and in case the manual how-to to unlock logs 🤧 There is a limit to everything 😂 ... View more

@gabo  Java 17 enforces stricter TLS/SSL algorithm constraints than older Java versions. Also your Microsoft JDBC driver 1.3.2 is very old which may not support this. So try upgrade the JDBC Driver and test it or downgrade your java to 11 and test Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

@egko  Your Splunk Enterprise 8.1x is outdated, and Splunk Enterprise Security 7.0.1 is likely incompatible, causing the KV Store to fail (from "Ready" to "Failed") when ES loads. Upgrade Splunk Enterprise, then upgrade ES to a compatible version. #https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/9.0/upgrade-or-migrate-splunk-enterprise/how-to-upgrade-splunk-enterprise Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

If both indexers will be receiving HEC data (very likely) then both must have the same set of HEC tokens.  Do that by putting the tokens in the inputs.conf file in an app on the CM and deploying it in the cluster bundle. ... View more

@Anders333  The lookup file is created with the fields _time and test, then you run stats values(test) as testing. This produces a new field testing in the search results. Splunk lookup files are schema‑flexible. If later commands introduce new fields, splunk adds them as new columns, even if they’re empty for existing rows. If you need only testing field then write your outputlookup command after your stats. Eg: | makeresults | eval test = "this is a testing thing" | stats values(test) as testing | outputlookup append=false test.csv Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

@quangtran  Modify your transofms.conf [route_UAT] REGEX = ^uat-.* SOURCE_KEY = MetaData:Host DEST_KEY = _TCP_ROUTING FORMAT = uat_indexer ... View more

@nooproblems  Since you already have add-ons everywhere, does your event include a syslog header? Could you provide a sample log? To ensure the Palo Alto add-on functions properly, make sure the syslog header is not included in your log. If you have syslog header add below to your inputs.conf no_appending_timestamp = true Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

@luispulido  As @bowesmana  mentioned, metadata is not reliable always for this use case. I have seen metadata can become outdated if new buckets are created. So i always prefer to run  | tstats latest(_time) as lastSeen where index=<my_index> by host Regards, Prewin 🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks! ... View more

You can read/create dashboard using _audit logs, where all this info can be seen, else you can use the below query index=your_index sourcetype=your_sourcetype | eval hour=strftime(_time,"%H") | stats count by user hour | eventstats avg(count) as avg stdev(count) as std by user | eval zscore=(count-avg)/std | where abs(zscore)>2 OR hour<9 OR hour>17 ... View more

Hi @PrewinThomas  I do have the following data in the indexes.conf file: [volume:cold] path = /splunk-data/cold [volume:hotwarm] path = /splunk-data/hotwarm maxVolumeDataSizeMB = 4500000 [wineventlog] homePath = volume:hotwarm/$_index_name/db coldPath = volume:cold/$_index_name/colddb tstatsHomePath = volume:hotwarm/$_index_name/datamodel_summary thawedPath = /splunk-data/cold/$_index_name/thaweddb   Are there any logs that will tell if there was an issue in resolving these paths? ... View more

Thanks in advance for the help. I’ve created an eventtype="auth_successful" for VPN  logs and I’d like to create field extractions based on this eventtype. I tried using a stanza like this in props.conf: [eventtype=auth_successful] EXTRACT-fields = ... but it didn’t work. It seems Splunk doesn’t support applying props directly to eventtypes (like it does for sourcetype, source, or host). Can someone confirm if this is the case, and what’s the best way to scope extractions so they only apply to events matching this eventtype? Thanks! ... View more

Hi. What version of Splunk are you running? I ran into a bad bug with both Splunk Enterprise 9.3.7 and 9.4.5. The heavy forwarders sending to DNS load balanced indexers get TCPOUT blocked.  This bug does not appear to be on the known issues despite many attempts by me trying to get it added there. It does not happen with 9.2.4. The Splunk JIRA that was opened is SPL-288904  The bug is said to be fixed in the upcoming releases 9.3.9 , 9.4.7, 10.0.3, 10.1 Hopefully soon. A workaround is the setting of dnsResolutionInterval in outputs.conf dnsResolutionInterval = <integer> * The base time interval, in seconds, at which indexer Domain Name Server (DNS) names are resolved to IP addresses. * This is used to compute runtime dnsResolutionInterval as follows: Runtime interval = 'dnsResolutionInterval' + (number of indexers in server settings - 1) * 30. * The DNS resolution interval is extended by 30 seconds for each additional indexer in the server setting. * Default: 300 seconds (5 minutes)   Splunk had recommended we set dnsResolutionInterval =480 (tcpout blocked). I tried 1000 (also blocked).  I have set it to 10000 (ie 10,000) and after ~ 3 days this seems to be working.   ... View more

What would be the additional certificates in that case if I can add to avoid the issue? And where usually we can add those before starting installation? ... View more