So, in the frozen bucket events: 07-24-2014 01:30:51.609 +0200 INFO BucketMover - will attempt to freeze: candidate='/opt/SP/apps/splunk/splunk-6.0.1/var/lib/splunk/rest/db/db_1392823223_1392819715_1'because frozenTimePeriodInSecs=2419200 exceeds difference between now=1406158251 and latest=1392823223' The latest time should be the timestamp of the earliest event in the bucket. Are you receiving freeze events where now - latest < 7776000 ? ... View more

You can route to nullqueue based on patterns in the events you want to drop: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad The following would prevent any events with the string p_summation_net from getting indexed. props.conf: [vmware:sourcetype] TRANSFORMS-null = drop_p_avg, drop_p_summation transforms.conf: [drop_p_avg] REGEX = p_average_net_ DEST_KEY = queue FORMAT = nullQueue These would need to be placed on your indexers. ... View more

No, you don't need to specify it should pickup the default (as long as the default hasn't been changed). Yes, It should recurse through any subdirectories it finds that it has access to. ... View more

Correct, if you monitor c:/documents/testdata/ splunk should try to ingest any files dropped there that it can read. You can also use wild card: c:/documents/testdata/*.log To ensure you only pickup log files in that directory, if there is risk of other files you don't want read showing up in that file. ... View more

The query required depends on your data, and the field that contains the information you require. In the example: | lookup dnslookup clientip as dst OUTPUT clienthost as DST_RESOLVED The even generating search, is returning a dst field, which is being resolved the resolved value is being stored in the field DST_RESOLVED ... View more

and apparently a hash symbol means to bold on this platform, my bad. ... View more

You don't need to restart the service, you can simply reload the deployment server: /opt/splunk/bin/splunk reload deploy-server That will cause the deployment server to reload, accepting any configuration changes. ... View more

So, volumes provide a better way to manage space when its shared among indexes. We set a frozen time per index, then we set a maxVolumeDataSize for the volume that is set to ~90% of the storage partition size. When the storage hits 90%, Splunk will go through ALL indexes on the volume and start rolling buckets until the size is below 90%. Basically, we use the volume setting as the failsafe to ensure the storage doesn't get exhausted. We set high homepath and maxdb size per index, managing the total storage by the volume settings. This works for us, there are advantages/disadvantages to this approach. ... View more

Once you correct the entry in serverclass,conf the Mac device should no longer be in the wrong server class. However, once added to the correct serverclass, it will not attempt to update any apps on the Mac device that are not defined in the Mac's current serverclass. For example, if you defined serverClass A to get App SampleApp, and the Mac was accidently part of ServerClass A it recived SampleApp from the deployment server. You then removed the Mac from serverClass A, and put it in serverClass B. Since serverClass B is not aware of SampleApp...it will not attempt to add/remove or modify the app from the Mac when they connect as serveClass B for the first time. What you could do, is define SampleApp for serverclass B which you use for the Macs. reload deployment server. Then remove SampleApp from serverClass B and reload the deployment server. The deployment server should remove SampleApp from the systems that fall under serverClassB ... View more

So, /splunkdata used to be the hot/warm (homepath) If so, is the directory structure still /splunkdata/${indexname}/db or did you rename the db direcotries to colddb? If they are still named db and your indexes.conf reference colddb...that could be the issue ... View more

Do you have the Splunk process running as a non-root user, and does that non-root user have access to /splunkdata Do you see any errors in splunkd.log or your internal index referencing the cold location. Are there currently buckets in the cold location? ... View more

It appears you have SE Linux enabled, have you followed: https://github.com/doksu/selinux_policy_for_splunk ... View more

maxTotalDataSizeMB = 1945600 This indicates the total size of the index. If you are only seeing the issue on a few indexes, I would verify these settings. ... View more

It appears you are referencing the threatHash field incorrectly. SELECT [ReceivedUTC] AS [timestamp], [ePO_xxxxxxxxxxxx].[dbo].[EPOEventsMT].[AutoID], [ThreatName] AS [signature], [ThreatType] AS [threat_type], [ThreatEventID] AS [signature_id], [ThreatCategory] AS [category], [ThreatSeverity] AS [severity_id], [DetectedUTC] AS [detected_timestamp], [TargetFileName] AS [file_name], [AnalyzerDetectionMethod] AS [detection_method], [ePO_xxxxxxxxxxxx].[dbo].[EPExtendedEventMT].[TargetHash] AS [target_hash], . . LEFT JOIN [ePO_xxxxxxxxxxxx].[dbo].[EPExtendedEventMT] ON [ePO_xxxxxxxxxxxx].[dbo].[EPOLeafNodeMT].[AutoID] = [ePO_xxxxxxxxxxxx].[dbo].[EPExtendedEventMT].[TargetHash] target_hash in a different table, so the full path is needed. Also, you had a LEFT Join statement, but the ON statement didn't include the same table as you were joining. Those are the issues that stood out to me, without seeing your DB structure. ... View more

The other common reasons I am aware of are when the max index size is hit or max volume size. ... View more

If you were to dig into the web data model, you the search the selects the logs to populate the index: (`cim_Web_indexes`) tag=web If you download the CIM compatible add-ons for apache and IIS, and sourcetype your events to match the add-ons..they should get parsed and tag'd correctly, allowing them to populate the data model. ... View more

Looking at the error: License Manager: Failed to contact license master: reason='Unable to connect to license master: https://xxx.com:8089 Connect to=https://xxx.com:8089 C timed out; exceeded 30sec', first failure time=1526522560 (Wed May 16 22:02:40 2018 EDT). You could increase the connection timeouts for the license server in server.conf: http://docs.splunk.com/Documentation/Splunk/7.1.0/Admin/Serverconf ... View more

https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Monitorfilesanddirectorieswithinputs.conf Look under the monitor files and directories section: recursive = true|false You just need to ensure the splunk process has access to the files/directories ... View more

What I meant to say: dropped packets or connections any kind of network error blockage by firewall or switch ACL any other form of connection data Configure switches/routers/firewall to syslog to your splunk instance. Install the appropriate apps for the network devices used. You can install streams and capture the metadata, or configure netflow collectors and send to streams. All depends on what you have available and what you are trying to do. But getting the logs from you network devices is probably a good first step and will meet many if not all of your needs. ... View more

Splunk is a data tool, for it to help you with those issues, you would need to provide the information required to identify the issue. specifically I need to catch network errors for things like, dropped packets or connections You will need to define what you mean here, packets are dropped on networks all the time. any kind of network error blockage by firewall or switch ACL any other form of connection data ... View more

There are multiple factors to why a bucket rolls. Run the following search: index=_* component=BucketMover "will attempt to freeze" You should see event similar to: 07-24-2014 01:30:51.609 +0200 INFO BucketMover - will attempt to freeze: candidate='/opt/SP/apps/splunk/splunk-6.0.1/var/lib/splunk/rest/db/db_1392823223_1392819715_1' **because frozenTimePeriodInSecs=2419200 exceeds difference between now=1406158251 and latest=1392823223** These events show the reason the bucket were rolled. That would help pinpoint the root cause. ... View more

I believe, you need to look for any roles with the capability delete_by_keyword, which the role can_delete has by default, but can be added to any role. The following search will show any role with that capability: | rest /servicesNS/nobody/system/admin/roles splunk_server=local | rename title as roles | eval caps=mvjoin(capabilities, " ") | eval caps=mvjoin(imported_capabilities, " ") | table roles caps icaps | stats values(caps) as caps, values(icaps) as icaps by roles | search caps=*delete_by_keyword* ... View more