This is a new instance being installed by a contractor who posited that it's Linux best practice that root owns all mount points.  Internally, a security architect (not a Splunk admin) supported the 'Linux best practice' argument.  In all of my many years of experience with Splunk, or in training, I have never heard of setting root as the owner of mount points within the /opt/splunk folder tree.  So this is a difference of professional opinion and I'm looking for definitive, authoritative consequences of having root as the owner of the mount points.   Often it takes someone outside the organization to validate an item. ... View more

Thanks. Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder? (this is a search head in a cluster) As well, what about in the /opt/splunk/var/lib folder on an indexer.  Note that buckets are actually stored in /opt/splunk/var/lib/splunk/ ... View more

Thanks.  Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder?  (this is a search head in a cluster) ... View more

So here is my understanding and the way that I've got our on-prem instance configured. hot buckets are stored on a local flash array.  When the bucket closes, it keeps the closed bucket on the flash drive and writes a copy to the S3 storage.  The S3 storage copy is considered to be the 'master copy'.   I try not to use the term 'warm bucket', but instead use 'cached bucket'.  All searches are performed on either hot or cached buckets on the local flash array.  Cached buckets are eligible for eviction from local storage by the cache manager.  So if your search needs a bucket that is not on the local storage, it will evict eligible cached buckets, retrieve the buckets from S3 storage and then perform the search. The frozenTimePeriod defines our overall retention time.  We use hotlist_recency_secs to define when a cached bucket is eligible for eviction.  That is. buckets less than the hotlist_recency_secs age are not eligible for eviction.  Our statistics show that probably 90% of the queries have a time span of 7 days or less (research gosplunk.com for query).  Thus, by setting the hotlist_recency_sec to 14 days, we are ensured that the search buckets are on local, searchable storage w/o having to reach out to the S3 storage (which is slower). One last thing.  We need a 1 yr searchable retention.  However, we also need to keep 30 months total retention.  To accomplish this, I use ingest actions to the S3 storage.  Ingest actions will write the events in compressed json format by year, months, day, and sourcetype.   Hope this helps. ... View more

These visualizations looks great.  However, I'm on version 7.1.1 and I don't see the visualizations.  Is there any special configurations/conditions required to get them to display? ... View more

That script, I believe, moves frozen buckets to an S3 storage location.  However, I what I want is to use the ingest actions  (Settings / Data / Ingest actions) to write all raw events in compressed JSON format.  Ingest actions is per source type and with nearly 100 or more source types, manually creating one for each is onerous.   Ingest actions use RULESET in the props to define how to store on the Remote File System (RFS). ... View more

To clarify.  Splunk +9 provides 'ingest actions' to filter, mask and route data.  This can be done on a heavy forwarder or indexer.  Heavy forwarders would receive their configuration from a deployment server, but clustered indexers receive their configuration from the cluster manager (I guess stand-alone indexers could get their configurations from a deployment server too).  It was that in the class documentation, they only mentioned that 'ingest actions' were deployed through the deployment server and not that they would be deployed by the cluster manager if you had clustered indexers. ... View more

I may get slapped for this answer since Splunk technically says it's deprecated, but you could consider using the LightWeightForwarder.  The LWF disables the parsing & index queues on the HF, but gives one the HEC inputs and Python (for running apps such as eStreamer, dbConnect, etc) as an intermediate relay.   A LWF disables the web interface, but that can be turned back on in a higher precedence web.conf. I use a LWF in the DMZ which gives me all the above, but also provides a deployment server for assets in the DMZ. ... View more

I realize this is a very old post, but as I was browsing I didn't see an answer to your questions.  The best answer is probably to use a LightweightForwarders.  Yes, I know it's supposedly deprecated, but it does pretty much what you want.  That is, it sends cooked, but unparsed/unindexed data to the indexer.  It also gives you all of the functionality such as Python, HEC inputs, etc.  By default, the LightweightForwarder will disable the web interface, but that can be turned though the web.conf setting.  I use this configuration in my DMZ.  ... View more

Correlation rules from sources such as ESCU will have a filter macro.  For example, the rule "ESCU - Access LSASS Memory for Dump Creation - Rule" has a macro named "access_lsass_memory_for_dump_creation_filter".  You should modify the macro rather than the rule itself.  Sometimes rules are updated in the ESCU.  If one modifies the rule directly, the search will be saved in the 'local' folder and have precedence over an updated rule in the default folder. ... View more

Seems like Splunk should incorporate that feature; however, as far as I can tell they haven't. My work around has been to: 1. Run Splunk https on port 8443. 2. Install lighttp light web server that redirects http port 80 to https port 8443. 3. Run firewalld and redirect port 443 to port 8443. ... View more

I'm seeing something similar with my v9 intermediate forwarders.  I have not upgraded my indexer cluster to v9 yet (8.2.5) and was wondering if the forwarders being at a higher version than the indexer was causing the issues.  I'm seeing events from the problem relay servers such as:    Invalid ACK received from indexer=aaa.bbb.ccc.ddd:9997. Got unexpected ACK with eventid=53825 Are you seeing anything such as that?   index=_internal sourcetype=splunkd ack ... View more