Starting with a comment, because I haven't used the Java SDK much so am working off docs and intuition... But I wonder if you need to supply a timeformat through JobArgs#setTimeFormat(String) when you provide your formatted string. Maybe it's not defaulted as timeformat is in splunk ? ... View more

If you want to use a deployment server to distribute the app from Splunk cloud, to your forwarders then you need to set it up. As you're having Splunk Cloud monitor everything else, installing the app on the deployment server is an option as well. Now you don't actually need a deployment server, the purpose it serves is to push out configuration to forwarders. If you have other forms of automation and configuration management those could be used to serve the same purpose. This is getting into how to best architect and deploy Splunk in your environment which to sufficiently answer means going through a large number of variables and trade offs which isn't easy on a community site. Your sales engineer could likely help some but to answer in depth for production I would recommend Splunk's professional services. If however you're just trying things out (with a cloud sandbox for example) don't worry about a deployment server for now and just manually install the forwarder app. (Last time I played with the sandbox, there were instructions on how to do this where you downloaded the forwarder app) ... View more

Splunk cloud provides you an app that you need to put on all forwarders you want sending data to Splunk Cloud. Port information would be specified in that app. Splunk Cloud does not provide a deployment server, if you want one you would have to set it up yourself, but it would be a good way to distribute the aforementioned app. ... View more

Your search doesn't even do what you say. stats first(id) as id gives a single id it does not give all distinct values. Fixing that, map is still terribly inefficient and by default will bomb out if you ever fed it more than (by default) 10 values of id. Kicking off that many individual searches to get what could be pulled back in one search is terribly inefficient ... View more

Well excellent then! converted comment to an answer. ... View more

How to manage data in Excel and convert that data to CSV is a bit outside the scope of this support site (This is a site for community Splunk support). Doing a Google search on how to convert multiple Excel sheets turns up a number of options ranging from programmatic to pre-built utilities for such a conversion. I have not done such, therefore cannot recommend any one particular way. Once you have the data in CSVs there are a number of things you could then do with them and Splunk, indexing being the most straightforward, but you could also build lookups around each CSV and use various lookup commands with it. I suppose If you know enough python it may be theoretically possible to build an external lookup script to open your excel file and retrieve data that way. However I don't know of anyone who has built such for Splunk yet, while in theory it could be possible to build you may want to consider effort and risk vs reward. ... View more

Try again? Overlooking the fact that your sourcetypes are different from what was stated in the question. The first two segments of your search, fetches a single user's id (which doesn't help solve the problem of getting data for each user as was asked for in the question). You then use a map to kick off a search to find that id in the app opening sourcetype (which wouldn't actually qualify since the field here is user_id instead of id) to get a single opening result and only the fields that are present with the opening sourcetype (if I had to guess, name and email come a user profile not the app opening. and while they could be present assuming an automatic lookup as I mention below, presence of such would eliminate the need for querying the user_profiles sourcetype). Additionally while map is a useful command, its usefulness quickly degrades as its performance does as it kicks off a separate search for each result present immediately before the map command. (it warns you if you have too many but this can be really slow). ... View more

Based on your table there, is your field actually called Name or is it actually 'CI Name' ? ... View more

Also from re-reading your question, I'm not sure if this was clear or not, but Splunk Enterprise is licensed based on bytes indexed per day not total. Say you have a 100GB license. On day 1 you could index a 50GB log file, and 50GB of other data and still be within your license. On day two you can index an additional 100GB of data from anywhere and still be ok. Also, Splunk understands that backfilling and reindexing sometimes is necessary, so they let you go over your Enterprise license 4 times (Free license 2 times) in a rolling 30 day period, without any negative impact. but if you're going over that often you may want to look at if logs you're sending into Splunk are indeed valuable and if not follow the advice of the "Keeping the Junk out of Splunk" conf2014 talk. I definitely recommend watching this talk. ... View more

I actually presented at .conf last year about how our 4.3 standalone indexers to 6.x clustered migration went like this: Install the new 6.x cluster with all existing (at least those that we wanted to keep) indexes configured Take a downtime of both the old indexers and the 6.x cluster during which you'll: Roll hot buckets to warm on the old cluster (temporarily disabling /blocking forwarding may be necessary here) Copy old indexes to new indexers (If you're going from 1 to multiple indexers, you can spread out the indexes across the new cluster Ensure that the 6.x cluster has all of the existing index/parse time configurations distributed to the slave indexers Configure the old indexers to indexAndForward to the new indexers (ensure that the 6.x receiving ports are open) Bring back up the 6.x cluster followed by the old server(s) What you've essentially done here is copied the data so your fresh install is now an upgrade, and made it so that any data that comes into the old cluster is also replicated over to the new cluster. This enables a controlled migration of any forwarders that you might have (especially if other teams control said forwarders) to switch over from one to the other, and all of your users data is in both places, and they can compare how dashboards and reports look in old and new environments. Now note you do double index for any new data coming into the old indexers in this scenario (at least until you retire those indexes on the old indexer, or the entire old indexers), so work with your sales rep to let them know that you're doing this. But in this way all of the data retires out as normal (still non-replicated, just like the upgrade, but all new buckets will be replicated, even in the same indexes). If you have too much data to copy, then take sk314's suggestion (and Raghav2384's confirmation) of keeping the old indexers around as additional search peers until data retires also works, especially if you keep the same names of indexes across both of these options. However your timeline of when you want to reclaim the old hardware versus your retention time may also sway you one way or another. You are also correct, if you went the route of putting buckets in thawed then you would have to manually remove them when you no longer want them. ... View more

There is some excellent documentation on the docs about configuring LDAP through the UI and configuring through conf files. Specifically for Active Directory, theres a sample config in the documentation on authentication.conf which will give you some good settings to get started. ... View more

Glad to hear it helped! Converted my comment to an answer so it can be accepted. ... View more

Yep, there are a number of other factors that can affect rolling from hot to warm. Most commonly, if Splunk is restarted, then all hot buckets are rolled to warm. Another possible scenario is that there is a setting for the max number of open hot buckets, if you're taking in data with disparate enough timestamps Splunk could decide that it needs another hot bucket and if you already have (I think the default is three) then one of them gets rolled to warm. And many other possibilities as well 🙂 ... View more

The most important thing to remember is that freezing (and other transitions from hot to warm to cold) operate strictly on entire buckets. You are correct, for a bucket to qualify to be frozen based on event time the entire bucket needs to be past the frozenTimePeriodinSecs. Otherwise you would be removing data that doesn't yet qualify to be removed. With your settings a bucket could be created up to 21 days wide, that bucket would not freeze until 28 days after the last event or 49 days after the first event in the bucket. Now to configure freezing based on size of your index, you should consider two other primary settings around retention, namely maxDataSize and maxTotalDataSizeMB also in indexes.conf. The former defines how large in bytes a bucket can be, and the latter is how large the entire index can be. When Splunk goes to make a new hot bucket, if there is not approximately maxDataSize under maxTotalDataSizeMB, then the oldest bucket is frozen. (Like the others these too are set on an index by index basis, and have default values) Now regarding the question in your comment, that setting doesn't really apply here. If your free disk space reaches that level or less, new events stop coming in (potentially being dropped if your queues fill up). It doesn't trigger freezing, but of course when buckets qualify to be frozen it could release new events to come in by freeing disk space) ... View more

It is true that good practice says to put configuration into apps for reasons of making it easier to distribute, and control scope at search time. That said fields.conf is a search time configuration. It has nothing to do with where inputs.conf resides (being an input time configuration file). Putting fields.conf in MyApp makes it such that it only applies to searches being run in the context of MyApp (assuming that these stanzas are not being exported to the system context). It doesn't limit the scope of these stanzas to a specific sourcetype or index. ... View more

The Configuration File Precedence discusses how Splunk configuration files at different points of execution combine to make the configuration that's running on the system both during interactive and non-interactive times. (This goes into depth as to how configuration from /apps/ comes into play, and applies to all config files). System is just the easiest to understand. Additionally as you get into editing configuration files across apps btool becomes insanely useful for figuring out how all this fits together. ... View more

In terms of parsing events, you may see some gains if you can split events with a simple LINE_BREAKER regex and SHOULD_LINEMERGE=false as you essentially skip a step. That said however, if your logs aren't able to be split with a simple enough regex, you could wind up more time than using the other aggregation settings (but that'd be something to measure as you try things out). JSON formatted and other structured events (like CSV and IIS/W3C logs), you actually have another option to play with as well, could potentially offload the parsing to your Universal Forwarders, and eliminating search time parsing by using INDEXED_EXTRACTIONS and _json sourcetype. ... View more

It's checked at parse time, however it's an input time setting. Otherwise the UF would not be able to know if it was cutting multibyte characters in half or not as it sends chunks of data on to the parser. ... View more

CHARSET is an attribute set into the pipeline data at input time, the props.conf should be on your Universal Forwarders. References: http://answers.splunk.com/answers/132918/charset-does-not-effect-when-data-is-from-a-universalforwarder.html http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F http://wiki.splunk.com/Community:HowIndexingWorks ... View more

One correction, Oracle stopped shipping the JDBC-ODBC Bridge driver in Java 1.8. JDBC itself is still very much supported. Now I'm assuming by Impala you mean the Cloudera Product you may have a few options here (none of which I've personally tried, but am piecing together via Documentation. YMMV). First of course is to go get the JDBC drivers for Impala and Setup DBConnect with an unsupported database so you can then create connections to Impala. Depending on what you're doing with the data, also in the realm of just outside of supported, you may be able to fight the HadoopConnect App to get what you need (with that app, I really mean I mean fight). But in the realm of actually supported technologies have you considered looking at Hunk? ... View more