These errors are logged when Splunk is unable to find a time stamp in the event and falls back on something else such as the time stamp of the previous event from that source/host/sourcetype, or the current time. ... View more

In a distributed deployment environment, you would have to use the CLI remotely (with a -uri option) or direct REST API calls to create new indexes and reload the indexing configuration on remote indexers. And yes, this operation (index creation + conf reload) would need to be performed on each indexer. ... View more

Hi andyw. A few questions for you : - How do you measure the memory consumption of the forwarder? Which metric (Vsize? RSS?) shows the 2Gb figure? - How large is the directory $SPLUNK_HOME/var/lib/splunk/fishbucket on the forwarder? - How many records exist in your fishbucket? To find out, run : $SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/ -k ALL | wc -l ... View more

I believe that SPL-42854 would only be relevant if the queues on the forwarder are full. Also, it can only account for up to 400MB or so of excess memory usage. @andyk : there's an important question I forgot to ask : What metric are you looking at when you report that 4.3Gb memory usage figure? Is this RSS/working set size or virtual size? ... View more

That's disappointing. Do you know if the CPU usage of splunkd has gone down since you introduced ignoreOlderThan? Also, are there other inputs defined on this forwarder that could be responsible for the memory usage? ... View more

One of the first things I would suggest is to use ignoreOlderThan in inputs.conf in order to keep splunkd from iterating through files with a modification time that has fallen behind a certain time window : ignoreOlderThan = <time window> * Causes the monitored input to stop checking files for updates if their modtime has passed this threshold. This improves the speed of file tracking operations when monitoring directory hierarchies with large numbers of historical files (for example, when active log files are colocated with old files that are no longer being written to). * As a result, do not select a cutoff that could ever occur for a file you wish to index. Take downtime into account! Suggested value: 14d , which means 2 weeks * A file whose modtime falls outside this time window when seen for the first time will not be indexed at all. * Value must be: <number><unit> (e.g., 7d is one week). Valid units are d (days), m (minutes), and s (seconds). * Default: disabled. Beyond that, you would need to use external means to further restrict the number of files that are exposed to splunkd so that it doesn't have to create and maintain a large number of objects in memory. Note that using a whitelist or blacklist in inputs.conf to exclude some files from indexing still exposes them to splunkd for evaluation, which contributes to its resource consumption. If your directory structure and log file distribution allows it, try to define one file monitor stanza per directory that contains logs to follow (up to 50 or so is reasonable) and use recursive = false in inputs.conf to scope the tailing processor to those directories only. This advice is of course irrelevant if you actually have live logs you want to index in all 699 directories. UPDATE : ** After discussing this with one of our developers, it turns out that ignoreOlderThan doesn't prevent us from creating an in-memory object for every file the tailing processor sees on splunkd startup, it just sets some aside never to be queried from disk ever again. As a result, ignoreOlderThan will have a positive effect on splunkd CPU usage but most likely **not on memory usage. ... View more

I suppose that you want your command to work in a distributed search environment and be executed on each search peer. The answer is that you can keep your search command just on the search-head but you'll need to set streaming = true when defining your command in commands.conf. That will ensure that your command is distributed. Note that you still need to set streaming = true even if your command is not a streaming command but say, an event-generating command. ... View more

This should be possible if your key/value pairs are present in the raw data of your events. Let's take a hypothetical example, assuming that this is the raw data of one of your events : 20 Jan 2011 19:34:22 E-1=100 Purgamentum init, exit purgamentum Let's say that field name "E-1" needs to be translated to "Error message" and field value "100" needs to be translated to "Banjo infestation". The first thing we want to do is to extract the "E-1" and "100" as a field each, not as a key/value pair. Let's do this with a rex : | rex "\d2:\d2:\d2 (?<field_name_code>[^\=]*)\=(?<field_value_code>[^\s]*)\s So now we have two new fields : field_name_code = E-1 and field_value_code = 100 . This is where your lookup comes in (let's say it's defined in transforms.conf as "errorlookup"] to output two new fields : "field_name" which will be looked up based on "field_name_code" and "field_value" which will be looked up based on "field_value_code". | lookup errorlookup field_name_code field_value_code OUTPUT field_name field_value So we'll now have a new field named "field_name" with a value of "Error message" and another one called "field_value" with a value of "Banjo infestation". Now you want to create a new field that concatenates "field_name" and "field_value" with a known delimiter (let's say "=") : | eval error_kv=field_name."=".field_value We'll now have a new field called "error_kv" with a value of "Error message=Banjo infestation". The final step is to use a delimiter-based search-time field extraction scoped to the "error_kv" field in order to create a new field named "field_name" with a value of "field_value". The options of the kv command do not allow for this to be done inline, so we'll need to define our field extraction in $SPLUNK_HOME/etc/system/local/transforms.conf (or anywhere else that is adequate to your configuration) : [extract_error_kv] SOURCE_KEY = error_kv DELIMS = "=" And now, we can call this custom extraction inline using kv : | kv extract_error_kv In our example, this should result in the creation of the field "Error message" with a value of "Banjo infestation"! Part of this is theory, so I'm quite curious to know if it works in your environment. Please let me know! ... View more

The dedup search command has a "keepempty" option that might fit your use-case : keepempty Syntax: keepempty=<bool> Description: If an event contains a null value for one or more of the specified fields, the event is either retained (T) or discarded (default, F). As I understand it, you could use three consecutive dedups to : Retain the most recent event for each unique value of the "Call_ID_1" field and all events that don't have a "Call_ID_1" field : | dedup Call_ID_1 keepempty=T . Retain the most recent event for each unique value of the "Call_ID_2" field and all events that don't have a "Call_ID_2" field : | dedup Call_ID_2 keepempty=T . Retain the most recent event for each unique value of the "Call_ID_3" field and all events that don't have a "Call_ID_3" field : | dedup Call_ID_3 keepempty=T . In the context of your search command, this would look something like : index=prodcorr OR index=premed_bts OR index=premed_pbts | rename MSP_vqmcallid AS Call_ID_1 PBTS_ORIG_SIP_CALL_ID AS Call_ID_2 PBTS_TERM_SIP_CALL_ID AS Call_ID_3 BTS_ORIG_SIP_CALL_ID AS Call_ID_3 | dedup Call_ID_1 keepempty=T | dedup Call_ID_2 keepempty=T | dedup Call_ID_3 keepempty=T | transaction Call_ID_1,Call_ID_2,Call_ID_3 maxspan=1d keepevicted=true As for ignoring events which don't have a given field, you simply do this with a search term such as NOT field=* . ... View more

This is a known limitation of CHECK_FOR_HEADER in a distributed environment. From props.conf.spec : CHECK_FOR_HEADER = [true|false] * Used for index-time field extractions only. (...) * The field names are stored in etc/apps/learned/local/props.conf. * Because of this, this feature will not work in most environments where the data is forwarded. In your case, CHECK_FOR_HEADER writes the header-based field extractions in the learned app of the forwarder, information which is of course not available when you search from the indexer. Of the two work-arounds discussed here, my preference goes to ftk's (the second one listed) : 1 - Ingest a couple of files on the indexer to create the field extractions there for sourcetypes iis and iis-2 at the minimum. 2 - Disable CHECK_FOR_HEADER on the forwarder for this sourcetype, clean up the learned app and configure the delimiter-based field extraction manually in transforms.conf on the indexer or search-head. ... View more

@mfeeny1 : I fear that this is the same issue mentioned in the tutorial where "requireClientCert = true" in the indexer's inputs.conf causes the SSL communication to fail, regardless of how valid the certificates are. This is bug SPL-37637. Please open a support case if you want to have this assessed further and tracked. ... View more

@mfeeny1 : If all of your indexers have certificates signed by the same CA (which I imagine is the case), you'll simply need to make sure that they each have their own [tcpout-server://server:port] stanza and that "sslRootCAPath" in each of these stanzas points to a file containing the public key of the CA that signed the server certificates present on the indexers. sslCertPath does indeed refer to the certificate presented by the forwarder to the indexer, but note that due to a bug, that certificate cannot currently be checked by the indexer. Regardless of that, it still needs to be present! ... View more

Your whitelist and inputs.conf stanza seem legit. Are you sure that the events returned by the search were not indexed before the new whitelist came into effect? ... View more

I fear that the problem here is that on first time run, splunkd encodes the default password for the default server certificate used for the splunkd management port, and outputs that to $SPLUNK_HOME/etc/system/local/server.conf : # cat etc/system/local/server.conf [sslConfig] sslKeysfilePassword = $1$S6+KkTowacjF As you know, settings in etc/system/local will take precedence over any other instance of the same configuration key anywhere else. Which means that the sslKeysfilePassword that you set with deployment server will not be taken into account. You can correct this by deleting the sslKeysfilePassword entry in etc/system/local/server.conf and restarting splunkd, but this is not something that deployment client can do for you. ... View more

Do the encoded representations for server.conf/sslKeysfilePassword and outputs.conf/sslPassword match? To verify this, compare the output of "%SPLUNK_HOME%\bin\splunk cmd btool server list sslConfig --debug" and "%SPLUNK_HOME%\bin\splunk cmd btool outputs list tcpout-server --debug" ... View more

Aw, now we can't see what the initial problem was! Don't be mad but I'll revert your question to the version that shows them. ... View more

The 1st problem I can see here is that the source-based stanza [source:///Users/dmaislin/Desktop/EMC/fsc.txt] will take precedence over the sourcetype-based [emc] stanza. As per props.conf.spec : **[<spec>] stanza precedence:** For settings that are specified in multiple categories of matching [<spec>] stanzas, [host::<host>] settings override [<sourcetype>] settings. Additionally, [source::<source>] settings override both [host::<host>] and [<sourcetype>] settings. You should not have two stanzas here. Also : Your TIME_PREFIX should be : TIME_PREFIX = timeStamp=" ...and to reflect this, I would suggest that you change your time-stamp extraction regex in emc-epoch.xml with : <![CDATA[0x([A-Fa-f0-9]{8})]]> Your MAX_TIMESTAMP_LOOKAHEAD should be anchored to the end of TIME_PREFIX, not to the beginning of the event : MAX_TIMESTAMP_LOOKAHEAD = 16 ... View more