All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hello hello! I think what you are looking for here is the `transaction` command, but it can have some extra over-head.  I'll leave some examples here to see if they work for you. Since your requirem... See more...
Hello hello! I think what you are looking for here is the `transaction` command, but it can have some extra over-head.  I'll leave some examples here to see if they work for you. Since your requirement is simple, I suggest using the `stats` command instead of `transaction`. If you wanted to look at a specific EventID first and then another specific EventID after, `transaction` might be easier to implement. Version using `transaction`:   index=lalala source=lalala (EventID=4720 OR (EventID=4728 AND PrimaryGroupId IN (512,516,517,518,519))) | transaction UserName maxspan=5m | search EventID=4720 AND EventID=4728   Version using `stats`: index=lalala source=lalala (EventID=4720 OR (EventID=4728 AND PrimaryGroupId IN (512,516,517,518,519))) | stats values(EventID) AS EventIDs by UserName | search EventIDs=4720 EventIDs=4728 Edit: Fixing the code blocks.
I'm trying to create a search in which the following should be done:  - look for a user creation process (ID 4720) - and then look (for the same user) if there is a follow up group adding event (... See more...
I'm trying to create a search in which the following should be done:  - look for a user creation process (ID 4720) - and then look (for the same user) if there is a follow up group adding event (4728) for privileged groups like (512,516 etc.)    my SPL was so far like that:    index=lalala source=lalala EventID=4720 OR 4728 PrimaryGroupId IN (512,516,517,518,519)   BUT that way I only look for either a user creation OR a user being added as a privileged user. but I want to like both. I understand that I need to somehow connect those two searches but I don't know how exactly.     
Hi, After completing the upgrade from Splunk Enterprise version 9.3.2 to v9.4 the KVstore will no longer start. Splunk has yet to do the KVstore upgrade to v7 as the KVstore cannot start. We were al... See more...
Hi, After completing the upgrade from Splunk Enterprise version 9.3.2 to v9.4 the KVstore will no longer start. Splunk has yet to do the KVstore upgrade to v7 as the KVstore cannot start. We were already on 4.2 wiredtiger. The is no [kvstore] stanza in server.conf so everything should be default. The relavent lines from splunkd.log are:     INFO KVStoreConfigurationProvider [9192 MainThread] - Since x509 is not enabled - using a default config from [sslConfig] for Mongod mTLS authentication WARN KVStoreConfigurationProvider [9192 MainThread] - Action scheduled, but event loop is not ready yet INFO MongodRunner [7668 KVStoreConfigurationThread] - Starting mongod with executable name=mongod-4.2.exe version=kvstore version 4.2 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --dbpath C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --storageEngine wiredTiger INFO MongodRunner [7668 KVStoreConfigurationThread] - Using cacheSize=1.65GB INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --port 8191 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --keyFile C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --replSet 4EA2F2AF-2584-4BB0-A2C4-414E7CB68BC2 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses) INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCAFile C:\Program Files\Splunk\etc\auth\cacert.pem INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --tlsAllowConnectionsWithoutCertificates for version 4.2 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames WARN KVStoreConfigurationProvider [9192 MainThread] - Action scheduled, but event loop is not ready yet INFO KVStoreConfigurationProvider [9192 MainThread] - "SAML cert db" registration with KVStore successful INFO KVStoreConfigurationProvider [9192 MainThread] - "Auth cert db" registration with KVStore successful INFO KVStoreConfigurationProvider [9192 MainThread] - "JsonWebToken Manager" registration with KVStore successful INFO KVStoreBackupRestore [1436 KVStoreBackupThread] - thread started. INFO KVStoreConfigurationProvider [9192 MainThread] - "Certificate Manager" registration with KVStore successful INFO MongodRunner [7668 KVStoreConfigurationThread] - Found an existing PFX certificate INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCertificateSelector subject=SplunkServerDefaultCert INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidCertificates INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 INFO MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --noscripting WARN MongoClient [7668 KVStoreConfigurationThread] - Disabling TLS hostname validation for localhost ERROR MongodRunner [5692 MongodLogThread] - mongod exited abnormally (exit code 14, status: exited with code 14) - look at mongod.log to investigate. ERROR KVStoreBulletinBoardManager [5692 MongodLogThread] - KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details. WARN KVStoreConfigurationProvider [5692 MongodLogThread] - Action scheduled, but event loop is not ready yet ERROR KVStoreBulletinBoardManager [5692 MongodLogThread] - KV Store changed status to failed. KVStore process terminated.. ERROR KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=KVStore service will not start because kvstore process terminated ERROR KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed. ERROR KVStoreBulletinBoardManager [7668 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details. INFO KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Mongod service shutting down     mogod.log contains the following:   W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead. W CONTROL [main] Option: sslCAFile is deprecated. Please use tlsCAFile instead. W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead. W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead. W CONTROL [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead. W CONTROL [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead. W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release. W NETWORK [main] Mixing certs from the system certificate store and PEM files. This may produced unexpected results. W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. W NETWORK [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting W ASIO [main] No TransportLayer configured during NetworkInterface startup W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. W ASIO [main] No TransportLayer configured during NetworkInterface startup W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. I CONTROL [initandlisten] MongoDB starting : pid=4640 port=8191 dbpath=C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=[redacted] I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2 I CONTROL [initandlisten] db version v4.2.24 I CONTROL [initandlisten] git version: 5e4ec1d24431fcdd28b579a024c5c801b8cde4e2 I CONTROL [initandlisten] allocator: tcmalloc I CONTROL [initandlisten] modules: enterprise I CONTROL [initandlisten] build environment: I CONTROL [initandlisten] distmod: windows-64 I CONTROL [initandlisten] distarch: x86_64 I CONTROL [initandlisten] target_arch: x86_64 I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, tls: { CAFile: "C:\Program Files\Splunk\etc\auth\cacert.pem", allowConnectionsWithoutCertificates: true, allowInvalidCertificates: true, allowInvalidHostnames: true, certificateSelector: "subject=SplunkServerDefaultCert", disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireTLS", tlsCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "4EA2F2AF-2584-4BB0-A2C4-414E7CB68BC2" }, security: { javascriptEnabled: false, keyFile: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 1.65 } } }, systemLog: { timeStampFormat: "iso8601-utc" } } W NETWORK [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. W NETWORK [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1689M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. F STORAGE [initandlisten] Reason: 129: Operation not supported F - [initandlisten] Fatal Assertion 28595 at src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp 928 F - [initandlisten] \n\n***aborting after fassert() failure\n\n    Does anyone have any idea how to resolve this? Thanks,
See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013
See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013
See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013
See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013
Things have improved a lot thanks to tsidxWritingLevel enhancements. If you set tsidxWritingLevel=4, the maximum available today, and all your buckets have been already written with this level you ... See more...
Things have improved a lot thanks to tsidxWritingLevel enhancements. If you set tsidxWritingLevel=4, the maximum available today, and all your buckets have been already written with this level you can achieve a compress ratio of 5.35:1 This means 55 TB of raw logs will occupy around 10 TB (tsidx + raw) on disk. At least this is what we have in our deployment. This number can vary depending on the type of data you are ingesting. Here the query I used, running All Time, starting from the one present in the Monitoring Console >> Indexing >> Index and Volumes >> Index Detail: Instance   | rest splunk_server=<oneOfYourIndexers> /services/data/indexes datatype=all | join type=outer title [ | rest splunk_server=<oneOfYourIndexers> /services/data/indexes-extended datatype=all ] | `dmc_exclude_indexes` | eval warm_bucket_size = coalesce('bucket_dirs.home.warm_bucket_size', 'bucket_dirs.home.size') | eval cold_bucket_size = coalesce('bucket_dirs.cold.bucket_size', 'bucket_dirs.cold.size') | eval hot_bucket_size = if(isnotnull(cold_bucket_size), total_size - cold_bucket_size - warm_bucket_size, total_size - warm_bucket_size) | eval thawed_bucket_size = coalesce('bucket_dirs.thawed.bucket_size', 'bucket_dirs.thawed.size') | eval warm_bucket_size_gb = coalesce(round(warm_bucket_size / 1024, 2), 0.00) | eval hot_bucket_size_gb = coalesce(round(hot_bucket_size / 1024, 2), 0.00) | eval cold_bucket_size_gb = coalesce(round(cold_bucket_size / 1024, 2), 0.00) | eval thawed_bucket_size_gb = coalesce(round(thawed_bucket_size / 1024, 2), 0.00) | eval warm_bucket_count = coalesce('bucket_dirs.home.warm_bucket_count', 0) | eval hot_bucket_count = coalesce('bucket_dirs.home.hot_bucket_count', 0) | eval cold_bucket_count = coalesce('bucket_dirs.cold.bucket_count', 0) | eval thawed_bucket_count = coalesce('bucket_dirs.thawed.bucket_count', 0) | eval home_event_count = coalesce('bucket_dirs.home.event_count', 0) | eval cold_event_count = coalesce('bucket_dirs.cold.event_count', 0) | eval thawed_event_count = coalesce('bucket_dirs.thawed.event_count', 0) | eval home_bucket_size_gb = coalesce(round((warm_bucket_size + hot_bucket_size) / 1024, 2), 0.00) | eval homeBucketMaxSizeGB = coalesce(round('homePath.maxDataSizeMB' / 1024, 2), 0.00) | eval home_bucket_capacity_gb = if(homeBucketMaxSizeGB > 0, homeBucketMaxSizeGB, "unlimited") | eval home_bucket_usage_gb = home_bucket_size_gb." / ".home_bucket_capacity_gb | eval cold_bucket_capacity_gb = coalesce(round('coldPath.maxDataSizeMB' / 1024, 2), 0.00) | eval cold_bucket_capacity_gb = if(cold_bucket_capacity_gb > 0, cold_bucket_capacity_gb, "unlimited") | eval cold_bucket_usage_gb = cold_bucket_size_gb." / ".cold_bucket_capacity_gb | eval currentDBSizeGB = round(currentDBSizeMB / 1024, 2) | eval maxTotalDataSizeGB = if(maxTotalDataSizeMB > 0, round(maxTotalDataSizeMB / 1024, 2), "unlimited") | eval disk_usage_gb = currentDBSizeGB." / ".maxTotalDataSizeGB | eval currentTimePeriodDay = coalesce(round((now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) / 86400, 0), 0) | eval frozenTimePeriodDay = coalesce(round(frozenTimePeriodInSecs / 86400, 0), 0) | eval frozenTimePeriodDay = if(frozenTimePeriodDay > 0, frozenTimePeriodDay, "unlimited") | eval freeze_period_viz_day = currentTimePeriodDay." / ".frozenTimePeriodDay | eval total_bucket_count = toString(coalesce(total_bucket_count, 0), "commas") | eval totalEventCount = toString(coalesce(totalEventCount, 0), "commas") | eval total_raw_size_gb = round(total_raw_size / 1024, 2) | eval avg_bucket_size_gb = round(currentDBSizeGB / total_bucket_count, 2) | eval compress_ratio = round(total_raw_size_gb / currentDBSizeGB, 2)." : 1" | fields title, datatype currentDBSizeGB, totalEventCount, total_bucket_count, avg_bucket_size_gb, total_raw_size_gb, compress_ratio, minTime, maxTime freeze_period_viz_day, disk_usage_gb, home_bucket_usage_gb, cold_bucket_usage_gb, hot_bucket_size_gb, warm_bucket_size_gb, cold_bucket_size_gb, thawed_bucket_size_gb, hot_bucket_count, warm_bucket_count, cold_bucket_count, thawed_bucket_count, home_event_count, cold_event_count, thawed_event_count, homePath, homePath_expanded, coldPath, coldPath_expanded, thawedPath, thawedPath_expanded, summaryHomePath_expanded, tstatsHomePath, tstatsHomePath_expanded, maxTotalDataSizeMB, frozenTimePeriodInSecs, homePath.maxDataSizeMB, coldPath.maxDataSizeMB, maxDataSize, maxHotBuckets, maxWarmDBCount | search title=* | table title currentDBSizeGB total_raw_size_gb compress_ratio | where isnotnull(total_raw_size_gb) | where isnotnull(compress_ratio) | stats sum(currentDBSizeGB) as currentDBSizeGB, sum(total_raw_size_gb) as total_raw_size_gb | eval compress_ratio = round(total_raw_size_gb / currentDBSizeGB, 2)." : 1"        
Hi another guys already show to you how you can technically add _time or event_time into your stats command. But I think that much more important thing is to understand and decide what _time you are... See more...
Hi another guys already show to you how you can technically add _time or event_time into your stats command. But I think that much more important thing is to understand and decide what _time you are needing with stats? Usually stats is used to make some statistics aggregations for some values or another option is join some data together. In both of these case you must understand your data and what you really want to show with your _time field value. I suppose that in many cases it's much harder to make decisions what _time you need to use. Use _time from one event, make some span e.g. 1min, 1hour etc. of calculate average or median time of events or something else? r. Ismo
The stats command discards all fields not mentioned in the command so, in this case, only the count, user, ip, and action fields are available.  Fields cannot be re-added after they've been discarded... See more...
The stats command discards all fields not mentioned in the command so, in this case, only the count, user, ip, and action fields are available.  Fields cannot be re-added after they've been discarded by such a command.   The solution is to include the desired field(s) in the stats command. | stats count by event_time, user, ip, action This may or may not make sense depending on your data and the desired output.
It's a FYI for all using 3rd party. Chances are many not paying attention on subsecond field value.
That's right. Go to third party to fix the issue if possible. 
So, the solution is to perhaps use a different third party solution or raise defect with said third party and get them to fix their data corruption?  (Not a Splunk problem!?)
Are you trying to perform the stats by _time also ? Just add your event_time into the stats command.  Change the event_time format to only Hour and Minute or just by hour ? index=* | eval event_t... See more...
Are you trying to perform the stats by _time also ? Just add your event_time into the stats command.  Change the event_time format to only Hour and Minute or just by hour ? index=* | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | stats count by event_time, user, ip, action | iplocation ip | sort -count
Hi @hcelep , after the stats command, you have only the fileds in the command, in your case: count, user, ip and action. If you want alto the _time, you have to add it to the stats command. You ha... See more...
Hi @hcelep , after the stats command, you have only the fileds in the command, in your case: count, user, ip and action. If you want alto the _time, you have to add it to the stats command. You have two methods to do this: add it to the BY clause,  choose the first or the last value for the groups. in the first case, remember to group the timestamps using the bin command: index=* | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | bin span=1h _time | stats count by _time user ip action | iplocation ip | sort -count in the second case, taking e.g. the first occurrence: index=* | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | stats count earliest(_time) AS _time BY user ip action | iplocation ip | sort -count Ciao. Giuseppe
Hey,   I want to add _time column after stats command but I couldn't select the best command. Forexample;   index=* | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | stats count by use... See more...
Hey,   I want to add _time column after stats command but I couldn't select the best command. Forexample;   index=* | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | stats count by user, ip, action | iplocation ip | sort -count   How can I add this field?   Thanks    
There is no issue if it's HF==>IDX. HF directly sending to IDX. It happens if HF==>(popular 3rd party s2s)==>IDX. HF sending to 3rd party over s2s and 3rd party sending to splunk indexer over s2s. ... See more...
There is no issue if it's HF==>IDX. HF directly sending to IDX. It happens if HF==>(popular 3rd party s2s)==>IDX. HF sending to 3rd party over s2s and 3rd party sending to splunk indexer over s2s.
Hi Basically you shouldn't uninstall previous versions when you are upgrading. If you uninstall it first then you are losing also fishbucket db which keep track ingested events. Basically that means... See more...
Hi Basically you shouldn't uninstall previous versions when you are upgrading. If you uninstall it first then you are losing also fishbucket db which keep track ingested events. Basically that means reinvesting all files etc. When you are updating UF version you should follow the same version path as defined for full enterprise. And remember to restart after every version update as otherwise e.g. DB conversions hasn't done. As @gcusello said there is a new feature which allow update those UF binaries. Currently it's a beta or restricted to some customers. You could see more from voc.splunk.com. If you have some other system management tools, then you can use those to update binaries, but again you must add needed steps to those workflows. r. Ismo
yes, this: spath input=msg query{} is probably just some cheating to see some data, but it does not actually work. It is just some overcomplicated syntax to see the whole array, but not individual ... See more...
yes, this: spath input=msg query{} is probably just some cheating to see some data, but it does not actually work. It is just some overcomplicated syntax to see the whole array, but not individual items; if the array would have more items, we won't get first item. If I need to get first item in array to work further, this just does not work
how works