using the following logs: 2015-06-30 15:23:56,286 UTC [14] INFO 14 Logger - [52365] - Activate Project=AAAA, START at 15:23:56.286 2015-06-30 15:23:56,288 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating Destination DPM Objects 2015-06-30 15:23:56,704 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager UserName svcDIMdpm 2015-06-30 15:23:56,705 UTC [14] DEBUG 14 Logger - [52365] - (queueProject) DpmInfoManager WebServiceURL http://KOSLOMRI033/SPSSMR/ActivateWebService/ActivateWebService.asmx 2015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - Activate Begin Activate 'AAAA' 2015-06-30 15:23:57,401 UTC [14] INFO 14 Logger - [52365] - AAAA:: Creating ActivateClient 2015-06-30 15:23:57,404 UTC [14] INFO 14 Logger - [52365] - ActivateClient UseWebService to True 2015-06-30 15:23:57,405 UTC [14] INFO 14 Logger - [52365] - ActivateClient Begin Activate 'AAAA' 2015-06-30 15:23:59,797 UTC [14] INFO 14 Logger - [52365] - ActivateClient Activate has been called 'AAAA' 2015-06-30 15:23:59,809 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA : TaskId=207f21a3-f030-4ec2-a5c5-625ccefa2b04, SUCCEEDED 2015-06-30 15:23:59,810 UTC [14] INFO 14 Logger - [52365] - Activating Project=AAAA, END at 15:23:59.810 2015-06-30 15:23:59,812 UTC [10] INFO 10 Logger - [52365] - Post Client URL will use default credentials 2015-06-30 15:23:59,813 UTC [10] INFO 10 Logger - [52365] - Data has been posted to http://poet-na601.grpitsrv.com/POET/KO/AutoActivateLogger.asmx/Log 2015-06-30 15:23:59,829 UTC [10] INFO 10 Logger - [52365] - AfterActivationProcess called for project AAAA 2015-06-30 15:23:59,830 UTC [10] INFO 10 Logger - [52365] - Activation EndInvoke complete for project AAAA The following search string appears to generate what you want. sourcetype="aas" ("Activate" OR "Activation" OR "Preview" OR "Live") ("START" OR "Begin" OR "EndInvoke") | eval ActivationType=case(searchmatch("Activate OR Activation"), "Activate", searchmatch("Preview"), "Preview", searchmatch("Live"), "Live") | rex "Begin\s\w+\s'(?<Project>[^']+)" | rex "project\s(?<Project>[^\s]+)" | eval DateTimeQueued=if(searchmatch("Begin"), _time, "") | stats earliest(_time) as StartTime min(DateTimeQueued) as DateTimeQueued latest(_time) as EndTime by Project ActivationType | fieldformat StartTime=strftime(StartTime, "%c") | fieldformat EndTime=strftime(EndTime, "%c") | fieldformat DateTimeQueued=strftime(DateTimeQueued, "%c") Approach is to ignore transactions, and instead identify the specific events we care about, and classify them with fields. Then we can just use stats on those fields to group things up. This assumes that Project is a unique identifier and only shows up once for each step. Recommend building that search pipe by pipe to see how it progresses. ... View more

Indexer Clusters: http://docs.splunk.com/Documentation/Splunk/6.2.4/Indexer/Manageappdeployment Search Head Clusters: http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/PropagateSHCconfigurationchanges ... View more

Try this. It gives you all events within one standard deviation of the average event size: ... | eval event_len=len(_raw) | eventstats avg(event_len) as avg_size stdev(event_len) as stdev | eval lower_bound=avg_size-stdev | eval upper_bound=avg_size+stdev | where event_len>lower_bound AND event_len<upper_bound eventstats generates aggregates, but applies them to individual events. So every event is retained, but in this case also gets fields added for avg_size and stdev . You can then compare the events length to those aggregates and filter accordingly. ... View more

SplunkAppforAWS is just the UI portion of the app. It has all the saved searches and dashboards. The inputs are part of the related Add-on ... View more

You can use index-time transforms to rewrite the event before it's written to the index. But you lose the prepending data. In transforms.conf: [my_data_json_extraction] SOURCE_KEY = _raw DEST_KEY = _raw REGEX = ^([^{]+)({.+})$ FORMAT = $2 In props.conf: [my_sourcetype] KV_MODE = json TRANSFORMS-whatever = my_data_json_extraction The name of the transforms stanza can be whatever you want. It just needs to be unique. Same for the TRANSFORMS-foo bit in props. Just make the part after TRANSFORMS- unique. I highly recommend testing this locally before applying it to production data as it is destructive. Make sure you capture anything you need from the initial part of the logs before applying this. Reference the documentation for props.conf and transforms.conf for details. ... View more

You can count from the end by using negative values. mvindex(split(URL,"/"),-1) will get you what you want. ... View more

Couple things here: Instead of stats, why not dedup? index="_internal" cost_center=$LOB$ | dedup series | table series | rename series as "LOB Splunk Indices" This will give you one row per value, instead of a single multivalued field. You can turn a multi-valued field into explicit records with the mvexpand command. index="_internal" cost_center=$LOB$ | stats values(series) as series | mvexpand series | rename series as "LOB Spunk Indices" ... View more

Looks like the data summary is coming from metadata searches. Example: | metadata type=sourcetypes You can build your own searches based on the data that command provides. What people can see is based on their role permissions. They won't see data for indexes they don't have access to. ... View more

We're actually evaluating the same thing. Here are the quick bullet points I've come up with so far: C4 + EBS volumes: Pros: Persistent storage. (Can be detached and reattached to the same instance, or a new one) High IOPS. EBS volumes are SSD backed. Even without using Provisioned IOPS, you get very good numbers here. (We see approx 3000 random seeks/sec from a 5TB EBS store) Cons: (Comparatively) Low throughput. At best, you're going to get 500MB/sec read/write if you're using a c4.8xl. Thankfully, they're EBS optimized by default, which is a requirement, IMO. (otherwise your incoming data and disk IO are competing for throughput) In our experience, this has been a bottleneck, and a primary reason I'm looking at the d2's myself. Cost. EBS volumes are expensive, and they don't benefit from the kinds of savings you get from reserved instances. Half the RAM of the D2's. D2s: Pros: WAY more storage capacity for your dollar. Double the RAM of the c4's Much better throughput. Bonnie++ testing of a d2.4xl with the ephemeral in a raid 0 showed sequential read/write speeds up to 2 GB/s. Cons: Lower IOPS. IN my testing, bonnie++ showed random seek numbers in the 400-600/sec range. Much lower than the SSD backed EBS volumes. Ephemeral storage. Data is gone if the instance is stopped or destroyed. (It does persist through OS reboots however) If you're going to consider the d2's, you have to do so with index replication. Leverage the location awareness of the feature with AWS Availability Zones. (An indexer in each AZ configured such that a copy of each bucket exists in all of them) With the increased capacity of the d2 instances, it's much easier on the pocket to use replication. The nice thing about using the d2's with replication is it lets you be more flexible in scaling your environment in response to load (Which may or may not matter depending on your particular use case) You can add/remove indexers from the cluster to scale up/down as long as you give the cluster opportunity to recover when pulling instances. The cost differences can depend on the capacity you need and how you plan to buy it. The 70% discount on 1 year pre-paid d2 instances makes a BIG difference when comparing cost, too. So do the math on your particular needs. I've found this tool very helpful in determining what's needed. Splunk Sizing One thing I will recommend is, even if you don't use replication, set up your indexers in a cluster from the start. It will make your life worlds easier if you decide to change your replication settings in the future, and makes configuration management easier. ... View more

Just ran into this today. The issue appears to be that content saved prior to migration is not fully recognized by clustering. (Which makes sense. It wasn't created in the cluster, so it's not fully aware of it) So the trick is to get it to create the appropriate clustering metadata. Thankfully, this is very easy: Edit the item and save it. You don't even have to actually change anything. Just click Edit > Edit Description and then click Save on the popup. Then you should be able to modify the permissions. ... View more