The easiest way to free up some space is to manually move some directories to an alternate location. If you want to keep the data, make sure you have an archive volume to move it to, if you don't care about the data, you can simply delete it. As long as you move entire 'buckets' of data you shouldn't run into any trouble. A complete bucket is everything inside a directory with the name format - db_, eg - db_1250709009_1250705372_75 This format tells you the latest event in the bucket (timestamp1), the earliest event (timestamp2), and hence the bucket-span, and finally the ID which was assigned to it on completion. Each bucket ID within an index must be unique. Click here for a more in-depth explanation of how buckets work. If you want to configure Splunk to manage disk-space itself, then you will want to consider a retirement policy. ... View more

The Unix app is the culprit. Even though you have disabled the index where the data should be stored, the inputs are still running and the data is flowing through Splunk, until it gets to the indexing processor which will throw it away. Indexed data volume in the 4.0.x and earlier versions was calculated before the data was actually written to disk, so even though your Unix app data isn't being kept, it still counts. If you disable the inputs, or the entire app, the violations will cease. This has changed in the latest 4.1 release, and data volumes are now calculated as the disk is written to. ... View more

Summary indexing IS NOT free in the 4.0.9 & earlier versions. This changed in the latest releases, so for 4.0.10, 4.1.x and later, it IS free. The _internal index is not counted towards your overall volume If you have a search-head pre 4.0.10, and you want to run summary index searches on it, you will need an Enterprise license. For 4.0.10 and later, you can use the forwarder license in $SPLUNK_HOME/etc ... View more

The optimal log format is - timestamp key=value key=value key=value key=value key=value key=value key=value key=value You can have other delimiters in there too like , or : but that's pretty much a personal preference. If the keys and values are easily recognizable, Splunk will index and search as fast as you can write it out. ... View more

Do you mean 170 distinct index directories in $SPLUNK_DB? or 170 buckets across all of your indexes? There's no hard limit, it's all dependant on the total amount of data you are indexing, your disk and the efficiency of your indexing settings. If you are actively indexing data to every single index, then Splunk may have a hard time keeping up with all of the aggregation, sourcetyping, etc. If you look for messages containing 'blocked!!=true' in the _internal index, that will tell you if you are hitting any resource limitations. CPU time is one possible bottleneck, disk contention is another. If there are no 'blocked' messages, then that would indicate that your instance is happy and able to cope with the workload. If you're not indexing a high volume of data, I wouldn't expect Splunk to be complaining very much. ... View more

I think you're just a bit unclear around the terminology & concepts in play here, we only implement 'blacklists' at the file level, not the event level. So you can blacklist entire files or directories using an entry in inputs.conf, but for individual events it's a bit more involved. This would be more clear if you were familiar with the 3.x version of Splunk, back then it took both a props.conf entry AND a transforms.conf entry to extract a field from an event. With the current version you only need to do this if you are configuring an index-time field-extraction, which is basically what you're trying to achieve here. In the config below, we have your regular inputs.conf entry - Inputs.conf [WinEventLog:Security] disabled = 0 As you can see, nothing relating to blacklisting here. Next comes the props.conf entry - Props.conf [source::WinEventLog:Security] TRANSFORMS-nullQ= nullFilter and then the transforms.conf entry is where the actual REGEX and actions are defined - Transforms.conf [nullFilter] REGEX = Account\sName:[\s|\w|-]+\$ DEST_KEY=queue FORMAT = nullQueue So in the 3.x world, if you had a setting in props.conf that started with 'REPORT-'. that meant it was a search-time field-extraction. If it started with 'TRANSFORMS-', that meant it was an index-time extraction. In this case, we want the action to run at index time, so we call it using TRANSFORMS. In the transforms.conf entry, we apply the REGEX to every single event covered by the props.conf spec - WinEventLog:Security - so this is an expensive filtering method from a CPU perspective. If the REGEX matches, then the event is routed to the 'nullQueue', i.e. it's thrown away and never makes it into the index. This only affects events that match the REGEX, if it's not a match, then the event continues to the index as normal. All of the configuration steps around this is documented here ... View more

Ok, so my first answer was way off, so I'm changing it. Yes, that's allowed. I missed the slightly important fact that we're dealing with interleaved events here, as in event1 starts, writes a couple of lines, event2 starts, writes a couple of lines, event1 writes some more, event2 writes some more, event 1 completes then event 2 completes. Can Splunk re-order this so that events are separated and concatenated properly? No it cannot, Splunk reads log files as they are written, it cannot be configured to re-order data. However, it would be possible to write a scripted input to do this first, and then feed the data to Splunk. Alternatively, you could just index the data as single-line events and then use the 'stats' or 'transaction' commands to sort the various lines together in search output. The final option would be to change the way in which your processes write their log files. Instead of one line at a time, is it possible to write a whole event at once so they are not intermingled? Or even separate log-files would do the trick too. ... View more

You should apply a TZ offset to your data at index-time as described here. That way, Splunk will know what timezone your data is coming from, and it will display it to you relative to the timezone that your Splunk server is running in. So if you're indexing GMT data on a Splunk instance in PST, the timestamp in your event will be 09:00, but the Splunk _time field will display 01:00 ... View more

Hi there, you must be new to the world of Splunk, welcome! With your Enterprise license, you should have received a 'Welcome to Enterprise Support' document that should answer a lot of questions, I'll make sure to send a copy on to you. You can split your license across as many indexing servers as you want, but that's really only beneficial when you get up into moderate to high indexing loads, 100GB per day and above. Check out this deployment guide for sizing info Forwarders and Light Forwarders are really a configuration mode of a standard Splunk instance. They are pretty easy and straightforward to set up in a simple configuration, but they can get complicated when you start having specific data requirements. If the documentation here doesn't answer all of your questions, then likely what you need is some Q&A time with a Splunk techie to answer all of your questions. I also strongly recommend that you attend some of the Splunk Education classes. You should speak to your Account Rep about both of these possibilities. ... View more

The PDF report server is an app that runs on a Splunk instance, just like the Windows app or the Unix app. It doesn't require an Enterprise license directly, but in order to send alerts any Splunk server requires an Enterprise license. If your main indexing server is on a Windows machine, and you have PDF server is sitting on a separate Linux server, then just install the Forwarder License on that Linux instance. That's an Enterprise license that will enable all Enterprise features but the volume that comes with it is 1MB - so you can't index any data. Aside - PDF reporting for Windows is currently in Development and should be available soon ... View more

1) You're seeing this message because you indexed more data than your license allows, 5 times or more in the last 30 days. If the increase in volume is unexpected, use the searches on this page to help you figure out where this extra data came from. Searches on the '_internal' index should not be blocked 2) If you are an enterprise customer with a valid support agreement, in order to remove the error message and get all of your searches working again, you should file a case with Splunk Support and request a reset key. Make sure you have found and addressed the cause of the violations before you do this, because it's only going to happen again if you don't. If you are using a free license you will need to have 14 days of license compliance before searching is restored 3) You can also set up a search to run once daily and check on the license violation count, and alert you if it increases. The below search will only return results if the quota has incremented: index=_internal source=*license_audit.log LicenseManager-Audit | delta quotaExceededCount as quotadiff | stats first(quotadiff) as quotadiff | search quotadiff>0 You can use the instructions in the docs here to set this up ... View more

Between version 4.0.x and 4.1 Splunk underwent some major changes in both functionality and features. To make the new stuff work, a migration script has to be run on certain files to ensure that they are in the correct format/version for the changes to work. If you went from 4.0.10 to 4.1.1 and wanted to go back to 4.0.10, I'd say you would be in for a rough ride, unless you have a backup from your earlier instance. Going from 4.1.1 back to 4.1 should prove less tricky, but there's no guarantee that it will solve your problem. In fact, it's more likely that the same problem will persist, and you could also possibly suffer from the issues that were fixed in the 4.1.1 build. My advice would be to hold tight and contact your Support rep directly. It's likely that there's a small migration issue that can be fixed pretty quickly. ... View more

Hi Sean, One solution would be to configure each call to talk to Radius and return the role information required, you could then use cachetiming to make that info persist long enough to be useful for any subsequent authentication calls. You could also configure your initial call to Radius to add user & role info to a dictionary and then the subsequent calls can just read from there, but you would have to make sure that the dictionary is refreshed on login every time, to account for role changes. ... View more

This will give you most of what you're looking for - ./splunk show license On my test Splunk instance, this returns: Your session is invalid. Please login. Splunk username: admin Password: Current Daily Usage Amount: 0 Expiration date: 2010-04-22T14:42:04-0700 Expiration State: 7 License level: 100 MB Product: Enterprise License violations: 2010-04-20T07:22:20-0700 License violation #5 2010-04-20T00:06:36-0700 License violation #4 2010-04-19T00:00:56-0700 License violation #3 2010-04-18T00:03:50-0700 License violation #2 2010-04-17T00:03:20-0700 License violation #1 Max Violations: 5 Peak usage: 1563 MB Days remaining: 2 day(s) Violation Period: 30 ... View more