Hi!   @isoutamo looped me in b/c he knows I'm currently in an Azure environment that's doing ~1Pb/day First, are you using SmartStore to offload older events to blob storage?  If so, around 1TB/day you're going to want to start thinking about splitting up your cluster because Azure throttles blob upload/download.  That WILL cause latency problems.  And there's also a whole bunch of SmartStore tuning you'll need to consider to minimize cache thrashing.   If you're not using SmartStore then the math goes a completely different way. Generally, what instance types are you using?  We've evaluated the following and find them are more than capable at our scale dasv5-series  dasv6-series  lsv3-series  ebsv5-series  edsv5-series  edsv6-series  If you ARE using SmartStore keep in mind that theres no concept of hot/cold,  just local disk/remote store so some of the faster local NVME may not scale up to what you need for your local cache and going for systems that don't have that and instead can scale your attached disk for your local cache is the way to go. That was our situation which is why we chose instances that don't have local disk, but allow lots of disks to be attached. If you AREN'T using SmartStore, then you'll want to look at the other instance types and leverage the NVME local disk for hot/warm and teh attached disk as cold. Beyond that, it's just a matter of picking the right size of your instance types to meet your SF/RF needs and data Ingest/Search load.   SmartStore/blob storage is really the piece that makes Azure unique.  Let me know if you are using it and we can discuss how to go about splitting your storage account(s) and possibly splitting your cluster. ... View more

Thank you all for this thread!   I've been working on Splunk for close to 10 years and JUST ran into this today for the first time and thought I was losing my mind because I can't find any record of etc/system/default/deploymentclient.conf on any of my systems.     Thank you for confirming that I'm not the only one thinking this is really, really weird but seems "normal"   ... View more

Any ideas of how this needs to be adjusted for Splunk Cloud where edit_tcp is not allowed?    I've got a case where something is working under a role with sc_admin capabilities but I don't want the role is should really run under to have everything that is capable of.  Documentation is sparse ... View more

I don't have links but from personal experience, the Windows OS has a higher overhead so all the sizing recommendations are undersized compared to real-world behavior.  In our specific scenario, measuring CPU/Memory utilization in the late days of v5 (pre-clustering) we saved nearly half of available resources by moving from Windows to Linux (being able to cut out roughly half of our hardware and still seeing better performance).  There have been vast improvements in how Splunk uses available resources since then so I suspect the differences won't be so egregious using anything in the 7.x, 8.x or 9.x versions but I also would be shocked if Windows ever performs better than *nix given so much of Splunk development start with *nix as the assumed OS. ... View more

Is it only this one directory? or are there others that need to be identified/hunted down? The "for instance" is unhelpfully vague. ... View more

Thank you for the suggestions, I'll keep them in mind for when I get to the searching stage, but the part that's got me stopped is getting the data indexed in the first place. ... View more

I'm not aware of any way to force the lookup command to use more than one field on the input side but you could do two lookups to get 2 new columns: index=abc |lookup lookupfile col1 OUTPUTNEW col3 AS Match_col1 |lookup lookupfile col2 OUTPUTNEW col3 AS Match_col2 Then use the eval command to compare those 2 new columns: | eval combined=if(isnull(Match_col1,"null",Match_col2) This will create a new column called "combined" where if the value of Match_col1 is blank (ie: col1 did NOT produce an output of col3) it will display "null" otherwise (for those lookups where col1 DID match), it'll carry forward the value Match_col2, which is where looking up col2 also resulted in an output of col3 This is particular option has been tested on a dataset where both the "col1" and "col2" will always have at least one matching entry to "col3".... I'm not certain how it'll behave if you have entries where both "col1" and "col2" don't have any corresponding value on your table. I'd assume it will result in a null entry, but you should probably validate that rather than assume. ... View more

The comment made by somesoni2, recommending the bucket command worked like a charm but I can't accept it as the correct answer because it was posted as a comment. 😞 If it gets moved I'll accept it as a great answer. Thank you! ... View more

Recommend you read all the details here: http://docs.splunk.com/Documentation/Splunk/6.6.0/SearchReference/Lookup But in short, from your search: (index=abc| table col1 col3| lookup lookupfile col1 OUTPUTNEW col3) On the assumption that col3 doesn't already exist in your raw event data , you don't necessarily need to do the | table command before the lookup. There's nothing wrong with doing so, it just doesn't make a lot of sense if that column is just blank. If col3 does exist in your raw events, that's a whole other story and there's nothing at all wrong with the table command. But assuming you just want to lookup up col1 and output col3, you've got the basic lookup syntax. The tricks are: 1) Make sure that "lookupfile" matches the stanza in the transforms.conf file which specifies the location of the lookup table file. Depending on if you created a lookup definition or not (http://docs.splunk.com/Documentation/Splunk/6.6.0/Knowledge/Aboutlookupsandfieldactions) this value may require the .csv at the end or be a different name than your csv file. 2) Make sure that col1 and col3 from your events each match the exact field name (case-sensitive) of what you're referencing in the table. If there's any differences use AS clauses as specified from the doco under the "Syntax" section. So in the end I'd tend to do: index=abc| lookup lookupfile lookupcol1 AS eventcol1 OUTPUTNEW col3 | table lookupcol1 col3 OR index=abc | table eventcol1 | lookup lookupfile lookupcol1 AS eventcol1 OUTPUTNEW col3 Either should get you basically the same end results. To specifically answer your questions: 1) No, you can use any combinations of the fields from your event data to cross-reference to any number of columns from the lookup table. There's no need to use all, just what you need. You can also do multiple lookups to also get any other combination you might be interested in... lookup col2 output col4, or lookup col1 output col 3 and col2 2) You don't need to create col3 the lookup command itself creates it. Read the "Usage" section of the doco to explain when/why you'd prefer to use OUTPUT vs OUTPUTNEW either of those options. ... View more

I'm curious about overriding this behavior locally also... I have 27 dashboard panels I'm trying to build that are unique filterings of the same underlying search that contains 80,000+ results. I'd prefer to just run that underlying search once, then call the loadjob and add the filters. It seems to be working just fine when the filtered results are less than 10,000 but if/when it's over that it truncates. ... View more

Thank you for the quick reply, and for the warning! I'll be sure to take precautions to preserve the original and any changes that are made now! ... View more

Thanks for the suggestion. We're currently bumping our license limit so thank you for the warning about that particular complication. I'll also have to ponder this a bit because I want to keep the existing host/source information I just want to add the heavy forwarder as a troubleshooting/monitoring function. Not to mention the complications of managing separate props/transforms on each heavy forwarder.. I hadn't previously considered that added complexity. But you've given me a good place to start. Thank you! ... View more

It's a range of devices including routers, switches, firewall, various security and networking appliances so they're ending up in a half-dozen or so individual indexes. ... View more

Sorry! Didn't realize that my original post cut out some information regarding the file name. I've tried removing the -$time:%Y-%m-d$ as you recommend, leaving just $name$ I've also tried removing the entire default ($name$-$time:%Y-%m-%d$) and manually entering just a filename without the .csv at the end. And I've also tried removing the entire default, manually entering the filename with the .csv at the end. All of them end up with the default ($name$-$time:%Y-%m-%d$) as the results. ... View more

The comment by gpradeepkumarreddy ended up resolving the issue but I can't accept it as an answer because it was posted as a comment. 😞 Clicking on the "Show All settings" did indeed cause the "Advanced Edit" to become visible again, on each node that I went through that extra step. I'll also note that I've confirmed that the changes I made within the Advanced Edit window so far has replicated across all nodes. But the warning that they should be verified is greatly appreciated! I'm still trying to figure out all the "will replicate/won't replicate" minutia since clustering is still a new feature for us! Thanks for the reminder and the great suggestion! ... View more