Take your total daily ingestion rate, divide by the number of indexers, and multiply by 30. That’s what you should have per indexer. This of course assumes your working set is 30 days. If you can quantify the time span your average searches cover, you can adjust accordingly. Most data probably is “stale” after 7 days, though this of course depends on your use case. ... View more

Unfortunately the Splunk App for Unix and Linux does not provide any Monitoring Console level details into your Splunk infrastructure. It's more designed for general Linux information (CPU/Disk/Memory monitoring). Does this other team need full access to the reports, or just some of the information? Most of the graphs can be grabbed by doing an "open in search" at the bottom right of the graph. However, most of the data is in _internal which is by default only visible to admin. You could open that up to others, but then the question is, what are you trying to protect? The data, or the administration of the system? ... View more

You can use the universal forwarder. The docs are somewhat confusing, but the line that says "Both full instances of Splunk Enterprise and universal forwarders support local collection of printer subsystem information. If you have Splunk Cloud and want to monitor printer subsystem information, use the universal forwarder to consume the information and forward it to your Splunk Cloud deployment." Splunk Cloud is Linux based, so there's no reason you can't install the UF and send data to your Linux based indexers. I sent a comment to the doc team to clarify the documentation. ... View more

Note that while this solves the query about how to do the lookup, it's still a horrible way of doing an event count, especially in large environments. I ran an environment that got conservatively 4 billion events a day (still a relatively small environment compared to some of the massive ones out there). People doing stats count on their data caused so many issues (when you have 100's of users doing bad queries, it adds up). You really should learn tstats if you're doing any sort of count of events. ... View more

What are you trying to figure out? | stats count is not going to be particularly efficient as it's a transforming command and must run on the search head and so is relatively inefficient as all the data has to come back to that single search head (especially if you have a ton of data and a lot of indexers). You might be better off looking into the tstats command, which will be orders of magnitude faster than stats (it looks at the index metadata (tsidx files), not the actual data). See https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/tstats for info on tstats and how to format that query. Refer to https://docs.splunk.com/Documentation/Splunk/latest/Search/Typesofcommands for more info on types of commands and where they run. ... View more

You are WAY over provisioned for the amount of data you're ingesting. Not a bad thing, since your searches will be super fast, but you probably spent way more than necessary. For ES, you figure a single indexer can ingest 100GB/day with full ES accelerated data searches turned on. I've seen much more, so that's a conservative value. You don't say how many spindles you have, but you're probably just fine. With SAS 15k drives, as long as you have more than 5 spindles on a RAID 5 you'll be fine for cold data. I guess you also have write caching so I'd say you're fine. For the SSD, you don't need RAID 10. RAID 10 is for best performance, but SSD alone is way more than you need for your 100GB/day ingestion and search quantity (I'm guessing you didn't mean 100k adhoc searches, since that would mean people typing in the searches - perhaps you meant acceleration?) You could go to RAID 5 and increase your total disk availability with little to no performance issue, or reserve the extra drives for when you need more indexers. ... View more

Lots of questions before an accurate answer can be offered (and even then it'll just be a swag). How many indexers for your 2TB/day ingestion? What kind of disks for your RAID 5? Hardware or software RAID? How many scheduled & accelerated data searches on your data and how far back does the average search look back? How many users? etc... If you're only using 24 hrs of data for the majority of your ES searches, you certainly don't need SSD/RAID10 for 90 days. You can probably do 2-4 weeks at most. If you already have the disk, go with RAID 5 or 6 for your hot/warm - SSD is way more IOPS than you probably need. You should be fine with RAID 5 for cold since you're not changing the data. RAID 5 suffers most for writes but the way Splunk writes data to cold is in larger chunks. So there will be a burst of writes when a bucket is rolled to cold, and after that, no writes until the next bucket is rolled. If your bucket sizes are large (ie, auto_high_volume or 750mb or larger), you'll not have much issue with disk writes. Unless you use absolutely terrible disks, you should be fine with RAID 5. ... View more

http://strftime.net/ - although it has a few missing options you might find in strptime, it's a good way to play with time formats. And as an aside, if you're using rsyslog, comment out the $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat line. It'll set your timestamps to RFC3339 standard format... ... View more

For indexers, if you're using a master server, you really should use the master to control your configs. How you control your master is up to you - if you want to use helm, jenkins, whatever, that's fine - just use the Splunk commands to push those configs to the indexers. This ensures that the configs are consistent and you can also control reboots somewhat more easily. Search heads are similar - if you're using a search cluster, push your configs to the deployers using your tool of choice, and use the Splunk cli commands to push the configs out to the search cluster. Individual search heads can be configured any way you like. I have mixed feelings about using configuration tools to manage forwarders. I've found it's a lot harder to ensure configurations if you have any questions about your config management. I've seen chef deployments where there were 100's of non-compliant forwarders because of chef issues, and if they were all pointing to a deployment server, you would then have a centralized place to look at what Splunk is up to. Now is the deployment server perfect? Far from it; it has a lot of shortcomings. But in general it's the easiest choice to start out with. You might want a base config that you push to the forwarders for initial installation and then push the rest from a deployment server. Be careful in that type of scenario - you can accidentally have competing configuration management systems. Remember Splunk is mostly stateful (the master is completely stateless, but the other systems keep varying degrees of state) so things like containers, autoscaling groups in AWS, etc, where you swap in/out base systems is not really ideal with the current Splunk design. Maybe in the future, if they consider containerization in their design, then it will be easier to containerize. ... View more

I would keep all Splunk core directories in the same location. Splunk writes a lot of temp stuff, and performance can be hindered if you are splitting up your install into lots of small storage volumes. How to answer this depends on your underlying storage configuration - if you're mounting the same storage pool (ie, one big SAN), there's not much to be gained by doing separate volumes for each, and you end up copying data around unnecessarily between mount points. If you're doing tiered storage, then you can split hot and cold to tier the data. Unless you can accurately determine data volume (ie, quantity), then splitting indexes into multiple locations is a royal pain in the rear. You're best off with one big hot volume and all indexes go into it, and one big cold volume if you have tiered storage. (there are exceptions to that depending on lots of things, which I can get into if you need). What I did was create /opt/splunk/data and in there do hot and cold volumes in their own directories. Don't put it in /opt/splunk/var/log. That's the log directories for Splunk. Someone could inadvertently clear the data. Note that even if you create a separate data directory, Splunk will still store some index related data in /opt/splunk/var/lib. I put an RFE in for Splunk to write all data in only one directory so the application could be split from the data more easily - right now it's a mess of locations that can cause some serious performance issues depending on what your underlying storage is (for example, putting Splunk on a root volume with relatively low performance can cause problems even if you're writing your index data to other storage, or if it's a search head and Splunk is writing out a lot of temp files during searches). I wouldn't bother with a volume for frozen, unless you often unfreeze large amounts of data. I don't think I did once in five years but that's just the way we were doing our storage... Note: There's also something called volumes within Splunk. Look at the volumes stanza in the indexes.conf. ... View more

I would consider the following in your project: Security - what are the security aspects of the data, access controls, etc. If you are concerned about this, there should be no mixing of data across accounts. You can use VPC's to segregate data as well, depending on how paranoid you are about the boundaries and how good you are at AWS management. Data resiliency - do you want to preserve your data irrespective of what happens with the underlying AWS services? If so, create multiple VPC's in different AZ's and put one Splunk site in each. I wouldn't consider separate accounts as separate Splunk sites. If you're going multisite for redundancy within a cluster, then set up separate VPC's in different AWS AZ's or regions. Splunk sites don't have to map to multiple VPCs though if you don't want to route traffic and pay for that, but you won't gain from the segregation of copies of data in different locations. Again, depends on how paranoid you are about your data being safe (if an AZ goes down, you still have access to your data on the other site if you split your sites between AZs). The cost of inter-VPC traffic will be the same as inter-account traffic, so you're not going to save anything on that. 1 search head and 2-3 indexers doesn't sound like a lot for 5000+ employees, but of course depends on the amount of data being collected and who's actually using Splunk. If everyone in this "company" should have access to data from each organization, then going centralized would make sense. It really depends on the details around what the data is, who should have access to it, who shouldn't have access to it, and how secure you want that data to be (both in terms of resiliency and access security). There's a lot of info you didn't provide that would lead the answer into one direction rather than another... ... View more

What do you mean "huge volumes"? How large are your json objects (ie, how many characters per object, and how many levels deep are the objects), and are you sure that they are fully compliant json objects? Why do you have heavy weight forwarders on your source server, another load balanced HWF layer, and then indexers? What do you mean "index time extractions but that is also slower due to the volume"? Are you saying the search is slower? ... View more

You don't really need any IOPS unless you're doing a persistent cache since the only thing disk will be used for is writing logs. You'd be better off with a several systems than one (as you said, horizontal vs monolithic). Adding more cores or ram in this situation won't really do much for performance. The system won't be doing anything but taking in data and spitting it back out which doesn't require much CPU. I've seen a single UF push mid-100's of GB/day on a small system (it was either a 2 or 4 core VM). You don't need a lot of horsepower for these systems. Make sure you disable the throughput limits on the UF. By default this is only 256kbps. Set it to 0 to push data as fast as the system can go. One thing to look at is providing more memory for the various queues, since the default is only 512kb. Memory on the system will not be used for much, so making queues 10mb or more won't hurt system resources. Check your metrics.log for "blocked=true" and also look at "group=tcpout_connections" to see if you see issues. I'm not sure if you need to increase the # of fd (max_fd in limits.conf) - that might be something to keep track of. You can also add a second pipeline (parallelIngestionPipelines = 2 in server.conf). You will want more than one system since you're pushing to an unknown number of indexers on the other end. If you have just one forwarder, you'll have your data all being sent over a single connection and to only one indexer at a time. If you have, say, 10 intermediate forwarders, and say 5,000 internal hosts, then you'll have (on average) 500 hosts sending data to each forwarder. My rule of thumb has been at least one forwarder per indexer. By doing this configuration, note that you are also limiting the quality of the data distribution on your indexers. This may not be critical to you, but be aware that it can cause issues with things like Enterprise Security. ... View more

Not quite a complete and correct answer by mreynov. Splunk uses a lot of different things besides CRC for file monitoring (atime is not one of them). For example, it uses mtime and size to determine if it has indexed a file and for the ignoreOlderThan option. In general there is no need for atime on most systems unless you need to know if a file was accessed at a specific time. That's why most modern file systems in Linux use relatime as the default mounting option (so your atime is probably already not being updated more than once a day). So go ahead and mount as noatime if you need zero updates, or relatime if you want sane updates (when file is changed or once a day). ... View more

Hot and cold are distinctions to allow you to save money on storage since in theory cold doesn't need to have the performance that hot does. You can certainly use one VMAX filesystem for it, but that's fairly expensive storage. If you can make a second lower performance filesystem (lower redundancy, slower disks, etc), then you can save on the cost per TB. Other than that, no reason not to put them on the same filesystem. If you do send it to one filesystem, don't make two separate mounts, as the OS will end up copying files from one to the other, since it will not know it's the same filesystem. If it's one mount, it's just a pointer move when a bucket rolls from warm to cold. ... View more

Not exactly sure what you mean by "Linux Application Server", as there is no such generic thing. If you're looking for a particular process that's running on a linux system, if you have the Splunk_TA_nix installed (https://splunkbase.splunk.com/app/833/), you can turn on the ps input and that will list all the running processes. Or you can create your own scripted input to run to collect the specific information that you want. All depends on exactly what you are looking for. Please provide some more details so a better answer can be provided. Thanks. ... View more

It’s a shift in the default authorize.conf file. Originally the capability dispatch_rest_to_indexers was in the [default] stanza, and now it’s move to [admin]. You will need to add it to the roles you want to have that capability. ... View more

I know this is an ancient question but figured I'd put in an answer for others... http://docs.splunk.com/Documentation/Splunk/7.0.3/Troubleshooting/HTTPthreadlimitissues ... View more

Ah, bad html link interpreter - I edited the comment to put a space before the ). Thanks! ... View more

Take a look at the indexes.conf documentation for Splunk 7.0. There's a new feature (unsupported, hopefully out in 7.1?) called remotePath and storageType (look at the very end for an example). Automatically handles S3 and caching data back for searching. ... View more