We have built a considerable amount of logic using a combination of python and kvstore collections to categorise incoming data The custom command can be called after the root event by using | datamodel ... or | tstats ... values() but I'm not finding a way to call the custom command (a streaming version was also developed) as a calculated field in the datamodel so we can leverage its acceleration without a huge | tstats ... values() chain Any idea on how to do this ? ... View more

Trying to consume some seismic data which input has a timestamp expressed in epoch time, but a timezone offset field expressed in [-1200, +1200] Description Timezone offset from UTC in minutes at the event epicenter. the offset above has no correlation with the known props.conf TIME_FORMAT's timezones %z UTC offset in the form +HHMM Was thinking about setting up a custom datetime.xml but couldn't find any "extract=" variable there meant to grab offsets, couldn't find the python module which parses datetime.xml either How should we approach this? ... View more

What would be the best way to get DCOS Mesos/Marathon to forward their containers logging to Splunk? As our Devs told me, "Docker’s runtime (excluding Docker Images) is already deprecated and we are going to get rid of the Docker daemon with time. Therefore when working with logging we should focus only on what DCOS & Mesos provide" Being Mesos just a way to orchestrate Docker clusters I'm looking for a way to achieve the same as Docker's logging driver into Splunk's HTTP event Collector ... View more

Has anyone had some experiences zookeeping container logs into Splunk? I'm experiencing logging is not standardized across containers and thus ending with half a dozen logging structures going into: /var/lib/mesos/slave/slaves/(?<agent>[^/]+)/frameworks/(?<framework>[^/]+)/executors/(?<executor>[^/]+)/runs/(?<run>[^/]+)/{stdout,stderr} Bumped into logspout which appears to be used to aggregate logs. Is this the way to go? Any alternative suggestion? ... View more

We may be having performance issues as newly saved search time extractions are not working even after being successfully tested via the Field Extractor Sample example: "faQUF","2.3.7","False","2","4","9","1","N-281","PF","19800","India Standard Time","3.8.0.5","2016-11-03T07:19:17.000Z","2016-11-03T10:49:35.000Z","3.8.0.8","/x/api/v2/hosts/fUF","","None","Windows 7 Enterprise","Service Pack 1","64-bit","7x-5x-fx-0x-xx-xx","dcfb" the following props.conf on were set on the SH [fireye:hx:asset_inventory] DATETIME_CONFIG = INDEXED_EXTRACTIONS = csv KV_MODE = none NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Structured description = Comma-separated value format. Set header and other settings in "Delimited Settings" disabled = false pulldown_type = true EXTRACT-agentId,agentVersion,excluded_from_containment,stats_acqs,stats_alerting_conditions,stats_alerts,stats_exploit_alerts,hostname,domain,gmt_offset_seconds,timezone,src_ip,last_audit_timestamp,last_poll_timestamp,last_poll_ip,url,last_alert_id,last_alert_timstamp,os_product_name,os_patch_level,os_bitness,src_mac,md5 = \"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\" EXTRACT-agentId = ^"(?P<agentId>[^"]*) NOTES: ** Search was run on Verbose ** the extraction was tested first as belonging to its owner, and then shared globally ** Both the single EXTRACT-agentId as well as the composed fields one were tested separately, just kept the single one to exemplify even such a simple extraction is not working Using the job inspector I'm seeing a very quick key value extraction (the 6 invocations may be the 6 default interesting fields Splunk extracts) Duration (seconds) Component Invocations 0.01 command.search.kv 6 I can only see the expected fields when I use the very same regex as a | rex command sourcetype = fireye:hx:asset_inventory | rex field=_raw "\"(?P<agentId>[^\"]*)\",\"(?P<agentVersion>[^\"]*)\",\"(?P<excluded_from_containment>[^\"]*)\",\"(?P<stats_acqs>[^\"]*)\",\"(?P<stats_alerting_conditions>[^\"]*)\",\"(?P<stats_alerts>[^\"]*)\",\"(?P<stats_exploit_alerts>[^\"]*)\",\"(?P<hostname>[^\"]*)\",\"(?P<domain>[^\"]*)\",\"(?P<gmt_offset_seconds>[^\"]*)\",\"(?P<timezone>[^\"]*)\",\"(?P<src_ip>[^\"]*)\",\"(?P<last_audit_timestamp>[^\"]*)\",\"(?P<last_poll_timestamp>[^\"]*)\",\"(?P<last_poll_ip>[^\"]*)\",\"(?P<url>[^\"]*)\",\"(?P<last_alert_id>[^\"]*)\",\"(?P<last_alert_timstamp>[^\"]*)\",\"(?P<os_product_name>[^\"]*)\",\"(?P<os_patch_level>[^\"]*)\",\"(?P<os_bitness>[^\"]*)\",\"(?P<src_mac>[^\"]*)\",\"(?P<md5>[^\"]*)\"" and as expected we get the rex command kicking in Duration (seconds) Component Invocations 1.40 command.rex 5,501 Anything which can point me to why this is broken? ... View more

What are the reasons which can cause this error in non clustered indexes ? As both the major and minor versions are the same between the SH and indexer (only the maintenance one is lower on the SH), according to the compatibility matrix it should work May this be due to a bundle replication issue? ... View more

Even though Splunk allows us to set a role level concurrent search jobs limit, it really does not allow us to ensure a role will have a minimum search jobs number allocated to it. We need a way to safeguard vital business critical dashboards/searches which becomes harder and harder without such a functionality (similar to Hadoop YARN queue capacity vs max capacity) As Splunk's usage in the organization grows more and more and from time to time we have rogue, poor quality dashboards absorbing a big % of our search capacity and the occasional scheduled PDF reaching our senior management with a graph hitting the max. Lowering all the roles concurrent search jobs limit doesn't really provide us the power we need as it would effectively promote resource waste as from time to time a role searches would be queued up even in situations where other roles searches are having minimal activity In cases of extreme concurrency, ad hoc searches from a role using a too high ratio between its minimum ensured 'capacity' and the search jobs limit should be killed to allow the minimum ensured capacity to be met across all roles Will raise this as an Enterprise enhancement case, but would still like to read some thoughts. ... View more