The only relevant number for "paying license usage" is "RolloverSummary" which is calculated once a day and written to "license_usage.log" and "license_usage_summary.log" by the License Manager.    ... View more

Hi, after one year I think it's time to give some feedback 🙂  Kidding, I was searching for something else and found this article by accident.  This information event_count=12146884, result_count=7, available_count=0, scan_count=2942980670 in the audit.log output will tell you that this search scanned almost 3 billion events on disk and just returned/used 12 million. So, 99.6% of the data read from disk was thrown away... sounds a little bit like boiling the ocean.  Looking at all the "searchmatch" patterns, you might put the TERMS() like "JwtTokenUtil" or "sendEmail"before the first pipe using  "() OR ()" because this will leverage the TSIDX Index (NOT the compressed raw events) and is 1000x faster than the full event scan you have done.  Another thought: You seem to mix a couple of sourcetypes in one index which is fine. Using the "sourcetype=" in the OR-Clause might also help to speed up searching because Splunk Enterprise will eliminate all events which are not matching the sourcetypes you want to see.  A shortened example: index=<index name> Error* ( (sourcetype=<st1> JwtTokenUtil) OR (sourcetype=<st2> SQLException) ... ) | eval ERROR_TYPE = case(searchmatch("SQLException"), "SQL Exceptions", searchmatch("JwtTokenUtil"), "JWTToken Error", searchmatch("JwtAuthenticationTokenFilter"), "Authentication error", searchmatch("java.lang.OutOfMemoryError: Java heap space"), "Out of memory error", searchmatch("sendEmail"), "Send Mail error", searchmatch("message was not delivered"), "IMAP error", searchmatch("sever not reponding"), "Server error", searchmatch("Timed Out"),"Timed Out Error", true(),"OTHER") | timechart count by ERROR_TYPE Will only return events with "Error*" and the two tokens I defined (with the right sourcetype).  This might save you from reading 99.6% of the events from disk, decompress them and do the field extraction for every single event...  Depending of the structure of your data, you might even use the "append" command and split the big search into multiple shorter ones but might avoid the "searchmatch" for all your 12 million results. index=<index name> Error* (sourcetype=<st1> JwtTokenUtil) | eval ERROR_TYPE="JWTToken Error" | fields _time ERROR_TYPE | append [ index=<index name> Error* sourcetype=<st2> SQLException | eval ERORR_TYPE="SQL Exceptions" | fields _time ERROR_TYPE ] | append [] ... | timechart count by ERROR_TYPE Try it.  You need to fix typos and maybe some syntax errors because I have written this by heart without testing in the "Search&Reporting" App. Hope this all make sense. Feel free to ask questions.   Greetings, Holger   ... View more

Just to link the topics... If you are running Splunk Enterprise in an environment where the difference between physical CPUs and virtual CPUs/threads is visible for SplunkD (e.g. bare metal, AWS, GCP, etc) you will see a higher "virtual" number which will be taken into account by the Splunk-internal engines.  See limits.conf.  If you are in a VMware environment and you see physical CPU=virtual CPU, you need to double the vCPU count in your VMware instance settings  to get the same performance because otherwise SplunkD will use less resources.  https://community.splunk.com/t5/Archive/How-does-Splunk-handle-Hyperthreading-in-a-VMware-environment/m-p/424582/highlight/true#M75311 ... View more

Just a short comment on the syntax checking in indexes.conf: Beware of typos in remote.s3.xyz!  A capitel "S" like "remote.S3.access_key" will be silently ignored and does not create an error message on startup like "storagetype" in the same file would!  The above examples are correct, just as a heads up.  ... View more

Hi, @jaracan, It's been a while since my deep VMware and Linux times... But IIRC, the lscpu will show the attribute bits which are visible, and because of "VT-x" you see "HT". But still only one thread per CPU.... maybe because of VMware doing things differently? No matter what... the information shown in Splunk is the important one for us.  And you can open a full can of worms using lscpu... https://unix.stackexchange.com/questions/33450/checking-if-hyperthreading-is-enabled-or-not#comment715791_230601 Greetings, Holger   ... View more

Hi @jaracan, the "loader" message in splunkd.log at startup or the Monitoring Console are showing the same information. http://<SPLUNK-IP>:8000/en-GB/app/splunk_monitoring_console/resource_usage_machine It will just report what the OS is presenting. Nothing special regarding the underlying virtual solution. If you have CPU=vCPU then most of the time you can be sure it's running in any kind of virtual environment because most of the servers are running with active HT nowadays.  But it depends on the virtual solution how they translate the HT "cores/threads" into the information which can be read from the OS. So, I would call it "educated" guess if you see CPU=vCPU. Does it make sense? Greetings, Holger   ... View more

Hi, nope. We do copy the buckets to SmartStore after rolling from hot to warm and mark them for eviction on the secondary Indexers. The cacheman is pretty much avare of almost every bucket.   BR, Holger ... View more

Hi, old question but I searched the forum today and want to clarify things... There are two points to mention: 1) Don't use "=" in the Search Filter https://docs.splunk.com/Documentation/Splunk/latest/Security/Addandeditroles#SPL_search_filter_syntax https://docs.splunk.com/Documentation/Splunk/latest/Security/Addandeditroles#Usage_of_search_filter_syntax_with_event_and_metrics_data Only index time field values are "safe"! 2) This does not work in Data Models! Take a look at the "Optimized Search" in the Jobinspector. With a "tstats count where index=_internal" you would see an additional "WHERE" which includes your Search Filter. If the field EventCode is in your Datamodel, you might think "Authentication.EventCode" in the Search Filter would do the trick but this is not the case! You would have to add this "DM-Name.EventCode" for each and every DataModel... not happening. So, it's simply not supported... HTH, Holger     ... View more

Hi, very old stuff but might be still a current problem... Have you ever tried to switch the file format from "parquet" to "orc"? parquet-hive-bundle-1.6.0.jar is f-uped... https://issues.apache.org/jira/browse/PARQUET-246 Looks like they fixed it in 1.8.0 which has never been shipped by Splunk Core... I have done my tests with Hadoop DMA using ORC... Worth a try? Good luck, Holger ... View more

Hi, just an update to make sure current options are set: v7.3+ https://docs.splunk.com/Documentation/Splunk/latest/Security/Ciphersuites HTH, Holger ... View more

Hi, just an update to make sure current options are set: v7.3+ https://docs.splunk.com/Documentation/Splunk/latest/Security/Ciphersuites HTH, Holger ... View more

Just an update to make sure people use the current options: (v7.3+) https://docs.splunk.com/Documentation/Splunk/latest/Security/Ciphersuites HTH, Holger ... View more

Hi, just for reference. You'll find a lot of shortcuts here: https://docs.splunk.com/Documentation/Splunk/7.3.0/Search/Parsingsearches Holger ... View more

Hi, this issue has been introduced with 7.2.0 and has been fixed in 7.2.4. https://docs.splunk.com/Documentation/Splunk/7.2.4/ReleaseNotes/Fixedissues SPL-162846, SPL-164977 HTH, Holger ... View more

Hi, this issue has been introduced with 7.2.0 and has been fixed in 7.2.4. https://docs.splunk.com/Documentation/Splunk/7.2.4/ReleaseNotes/Fixedissues SPL-162846, SPL-164977 HTH, Holger ... View more

Hi, looks like you found a solution: https://github.com/splunk/kafka-connect-splunk/issues/151 @Ken Chen said: Did you have valid CA signed certificate on your Splunk localhost ? If not, disable the cert validation ( hec.ssl.validate.cert to false) in the config and retry for the string converter scenario. HTH, Holger ... View more

Hi, old question but things have changed the last couple of years and I add this just for reference: We do have this kind of information now in the Monitoring Console and _introspection if you are running Splunk on the right OS. http://docs.splunk.com/Documentation/Splunk/latest/DMC/ResourceusageDeployment http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Abouttheplatforminstrumentationframework http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Whatdatagetslogged http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Whatdatagetslogged#I.2FO_statistics And because it's written in JSON format you are able to use "tstats" to get statistics fast. Would be great to mark this question/answers as "solved" Happy splunking, Holger ... View more

Hi, just found this discussion... I know you solved your problem, which is great (BTW: marking this discussion as answered would help others :-). The main part is the filtering of internal logs if you use an intermediate forwarder architecture, like you do. You found the black/whitelisting. Much easier would be: forwardedindex.filter.disable = true which if off (=false) by default. This is needed on the intermediate HF in your case. Hope to help others for future reference. Happy splunking, Holger ... View more

Hi, as mentioned in the TechBrief there are some things to take care of if you run Splunk Enterprise at scale. Rich7177's answers is correct for really small environments. Regarding HT, Splunk is able to detect this. Look for log lines like "Detected 32 (virtual) CPUs, 16 CPU core" in you splunkd.log. VMware e.g. will count every "virtual" core and at the end the best practices approach is to use 2x number of vCores compared to a bare metal setup. Meaning, instead of 12 cores for the reference architecture you'll configure 24 "vCores" for your VMware instance. HTH, Holger ... View more

Hi, just to make sure we are talking about the right thing: 100GB/day results in approx 15GB on disk (journal.gz). This data is streamed from the primary peer (indexer) to the secondary peer where it's indexed again (multi site cluster might do things differently). http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Bucketsandclusters So... 15GB/day is approx 1/6 megabyte/sec which is approx 1.5 MEGABIT/sec on the wire... And yes, peeks apply. Seriously, are we talking about bandwidth control in a datacenter where you have 1-40 GBit/sec? If you want to tweak the mentioned settings, please involve Splunk PS before you do... HTH, Holger ... View more

Hi, just to add one important thing (if someone finds this article using a search engine). In a normal clustered environment, a primary peer (indexer) is only streaming the compressed raw data (aka journal.gz) and some metadata, we estimate this volume with 15% of RAW. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Bucketsandclusters The secondary peer is then indexing the data again, depending on the search factor configured. Just to make sure we are talking about the right thing.... If an indexer is streaming 100GB/day of raw data (which is approx 15GB/day in journal.gz) this results in about 1/6th megabyte/sec on the wire... which is approx 1.5 MEGABIT/sec... I don't want to be rude... but Guys, really? Bandwidth control in a datacenter? For 1/600th or 1/6000th of your available bandwidth? HTH, Holger ... View more

Hi, there are some options... but remember Splunk is more like an event recorder. Since most of the fields are extracted at search time, we can't identify PII data at ingestion time. Easiest thing is to scrub the data at ingestion time... props/transforms.conf -> anonymize the data The "delete" command does not really delete, it marks "as deleted". You can set the retention time in the index accordingly... data will be removed completely. If you need to remove a specific event, you might dump the index via a search, remove the index and then re-index it... not nice but an option. Again, Splunk is not an RDBMS where you just delete a row. HTH, Holger ... View more

Hi, I have seen another error where this solution might be very helpful: If you install Splunk (with Splunk Analytics for Hadoop license, or just you want to use Hadoop Data Roll) on a separate SH and in addition are just using the "Hadoop Client Libraries" for your distribution (Apache Hadoop, Hortonworks HDP, Cloudera CDH, ...) you might need to copy mapred-site.xml and yarn-site.xml from a NodeManager or DataNode of your cluster to /path/to/your/Hadoop Packages/etc/hadoop/conf/ Otherwise you need to set a couple of different parameters in your Splunk Hadoop Provider (vix.yarn..., vix.mapred..., vix.mapreduce...) If index=hadoop-index | head 10 returns data... HDFS is working if (in SmartMode!!!!!!) index=hadoop-index | stats count returns a result without errors... you've done the right thing. And don't forget this cool bog post: https://www.splunk.com/blog/2014/05/14/hunkonhunk.html HTH, Holger ... View more