I'm building a search which takes a URL and returns all events from separate indexes/products where a client (user endpoint, server, etc) attempted access.  The goal is to answer "who tried to visit url X". I have reviewed the default CIM data models here: https://docs.splunk.com/Documentation/CIM/5.1.0/User/CIMfields However, none seem to fit this specific use case.  Can anyone sanity check me to see if I've overlooked one?  Thanks! ... View more

We are periodically seeing spikes of Storage I/O Saturation (Monitoring Console > Resource Usage: Deployment).  When split by host we can see that this is affecting all 6 indexers nearly simultaneously for the /opt/splunkdata mount points.  As expected, this triggers the Health Status notification throughout the day (warning or alert). To note, Load Averages are regularly > 5% with CPU usage normally under 10% for each indexer (24 cores each).  RAM usage around 30% per indexer.  We are wondering if our physical storage and/or network might be a bottleneck or if it's something on the Splunk side. For a Splunk Admin beginner, could someone please offer some suggestions on where we could start troubleshooting these spikes or explain in more detail the specifics around Storage I/O Saturation? We are on Enterprise 9.0.4 across the board and considering the recent update sooner than later. Thank you! ... View more

New customer seeking guidance for creating indexes/sourcetypes and determining granularity.  Primarily we're looking for deeper guidance on why more so than what.  We have a large, complex environment. Our naming scheme for indexes thus far is: organization_category_purpose (ex acme_net_fw) organization - unique to us, required, primarily used to segment data between organizations. category - broad, like network, application, endpoint, etc purpose - more specific, largely unique per category Does the following seem best practice, for firewalls? 2 or 3 indexes used by firewalls (traffic, operations, maybe threats?) Multiple sourcetypes split into the various indexes We are looking at SC4S as a guide (https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/PaloaltoNetworks/panos/) although their examples are not always consistent. We are struggling to determine how granular to be with the purpose of the index and with the amount of possible sourcetypes we can/will have.  We do not have the need to specify sensitivity or retention time.  Furthermore, we do not have the need to separate security/infrastructure teams. This slide from a Splunk presentation suggests that many sourcetypes get their own index for efficiency: Questions With 4-5 separate firewall products in use in one organization (the most complex), we're looking at 20-25 unique sourcetypes distributed into around 3 firewall indexes, just for firewalls.  Does this sound correct? We want to avoid unnecessary complexity for future searches, documentation, etc while not destroying our efficiency. Can anyone speak into their experiences with creating too many/too few indexes?  Specifically on long-term organization, search efficiency, overall experience? Can anyone offer any additional real-world guidance on creating a data catalog? We can't see any reason to split up windows event logs for endpoints (security/application, etc) but could see security being separate from the others for DCs.  Does that sound correct? Any resources or guidance appreciated.  Here's what we're using so far: SC4S example structure: https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/PaloaltoNetworks/panos/ https://lantern.splunk.com/Splunk_Success_Framework/Data_Management/Naming_conventions https://subscription.packtpub.com/book/data/9781789531091/5/ch05lvl1sec32/best-practices-for-administering-splunk https://kinneygroup.com/blog/the-proverbial-8-ball-splunk-implementation/ ... View more