Migration from a standalone searchhead to a SHC Here is the document that discusses how to migration from a standalone to a Search Head Cluster: http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/Migratefromstandalonesearchheads Add a Search Peer Here is the doc that discusses how to add search peers: http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/Configuredistributedsearch If you are adding multiple standalone indexers I would look at this section: http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/Configuredistributedsearch#Edit_distsearch.conf Essentially you would distribute the distsearch.conf from the Deployer and then distribute the key files which is covered in the following section: http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/Configuredistributedsearch#Distribute_the_key_files ... View more

Splunk on this host(s) is configured to forward its data to "local_55153" which is defined in outputs.conf: Forwarding to output group local_55153 has been blocked for 1550 seconds. If it is unable to connect to that output group, the data will begin to queue and once the queues are filled it will no longer process/ingest new data. You will want to review ouputs.conf on these hosts and determine if the settings are correct. If they are, then you will want to ensure that the instances that make up "local_55153" are reachable, are listening on the relevant port and are able to accept data. ... View more

As others have mentioned, you cannot alter data already indexed. If you correct the source type at the source so all new data utilizes the new sourcetype, the old data should eventually age out. You could also look at using an alias for the existing data with the wrong sourcetype: http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Renamesourcetypes ... View more

Here is a document that covers Forwarder Management Compatibility: http://docs.splunk.com/Documentation/Splunk/7.0.2/Updating/Forwardermanagementcompatibility#List_of_incompatibilities If you look in the table, for the setting stateOnClient, it states: Single per-app setting across all server classes In the notes section, below the table, it explains: Single per-app setting across all server classes: The **stateOnClient** and **restartSplunkd** attributes are configured under the app-level stanzas. App-level stanzas include both an appName and a serverClassName component, like this: [serverClass::app:]. See "Use serverclass.conf to define server classes". If you are not using forwarder management, you can configure those attributes differently depending on the server class, even for the same app. For example, in [serverClass:X:app:A], you can specify stateOnClient=enabled, while in [serverClass:Y:app:A] (same app, different server class), you can specify stateOnClient=disabled. So, depending on the server class, the same app will be either enabled or disabled when it gets downloaded to the client. However, forwarder management allows only a single definition for each app, across all server classes in which it is used. So, [serverClass:X:app:A] and [serverClass:Y:app:A] must both configure stateOnClient identically. The same condition applies to restartSplunkd. If you notice the 2nd paragraph, it says: If you are not using forwarder management, you can configure those attributes differently depending on the server class, even for the same app. So you can accomplish what you want, but you have to manage it through CLI. It is not supported in the gui. ... View more

Although this doc is specifically for Heavy Weight Forwarders, this should still work on an indexer: http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd#Forward_a_subset_of_data A very basic example would be something like: ################# # outputs.conf # ################# [tcpout] defaultGroup = myindexers [tcpout:myindexers] server = indexer01:9997 sendCookedData = true ############## # props.conf # ############## # capture data from sourcetype auditd [auditd] TRANSFORMS-auditd = myindexers ################### # transforms.conf # ################### [myindexers] REGEX = * DEST_KEY=_TCP_ROUTING FORMAT = myindexers This would route all data for that sourcetype to "myindexers". You could repeat that for each relevant sourcetype. Further, you can also filter that data further using regex to capture what you want and route the rest to nullqueue: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest ############## # props.conf # ############## # capture data from sourcetype auditd [auditd] TRANSFORMS-auditd = myindexers,nixnull ################### # transforms.conf # ################### [myindexers] REGEX = <some-regex-for-data-I-want DEST_KEY=_TCP_ROUTING FORMAT = myindexers [nixnull] REGEX = * DEST_KEY = queue FORMAT = nullQueue One important thing to keep in mind, with default settings in place, if the connection to the "myindexer" group is blocked, the queues will get blocked and it could result in no data being indexed on the customers indexers. If it is important to capture the data but acceptable to have some data loss if the connection drops, you could utilize the following setting in outputs.conf: dropEventsOnQueueFull = <integer> * If set to a positive number, wait <integer> seconds before throwing out all new events until the output queue has space. * Setting this to -1 or 0 will cause the output queue to block when it gets full, causing further blocking up the processing chain. * If any target group's queue is blocked, no more data will reach any other target group. * Using auto load-balancing is the best way to minimize this condition, because, in that case, multiple receivers must be down (or jammed up) before queue blocking can occur. * Defaults to -1 (do not drop events). * DO NOT SET THIS VALUE TO A POSITIVE INTEGER IF YOU ARE MONITORING FILES! For example: ################# # outputs.conf # ################# [tcpout] defaultGroup = myindexers [tcpout:myindexers] server = indexer01:9997 sendCookedData = true dropEventsOnQueueFull = 60 Again this is very basic, perhaps I haven't had enough coffee yet and there is an error in there but it should give you an approach to test and the documents related to that. --edit, and also I would keep the index name seperate, it is likely better to keep their data isolated instead of intermingled with yours. ... View more

You will want to look at the documentation here: http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest Essentially you will want to route the unwanted data to the nullqueue. ... View more

Glad its working for you. 🙂 ... View more

No you shouldn't need to copy it to local. Have you restarted Splunk on the SH since installing the APP? If so, you can look at btool to see if something is taking a higher precedence. From $SPLUNK_HOME/bin/ ./splunk btool authorize list --debug Do you see the settings? ... View more

I don't see any existing bug for the same crash for Splunk 6.1 or specifically 6.1.6 which is what I believe you have. I'd confirm, based on what I am seeing, that you have THP disabled. It is a memory management scheme that can cause problems when left enabled on instances running Splunk. This can include a variety of memory and stability issues and crashes: Transparent huge memory pages (THP) and Splunk performance. How do I disable Transparent Huge Pages (THP) and confirm that it is disabled? If that is already disabled, I would upgrade to the latest version of 6.1 which is 6.1.13, to eliminate any potential bugs you may be encountering. ... View more

Just an additional comment, if one wishes to utilize the dhFile setting in server.conf, on a windows host, Splunk may not start unless there are quotes (" ") around the setting. I added that to another post as well: https://answers.splunk.com/answers/525172/why-am-i-unable-to-use-dhfile-and-2048-encryption.html?childToView=550177#answer-550177 ... View more

Just in case someone runs across this post. The solution is to put quotes around the setting: dhFile = "$SPLUNK_HOME\etc\auth\splunkweb\DH2048.pem " This same solution was also provided on another answers post: https://answers.splunk.com/answers/525176/webconf-not-working-for-dhfile-in-2048-encryption.html I ran into a similar issue where if this was applied in server.conf without quotes, the splunk service would not start on Windows 2008 or 2012. A linux server was not affected. I have opened a bug to have this reviewed but wanted to comment here so others can see this. ... View more

Yes, I will keep you updated. ... View more

It does not cause any problems other than this WARN message. ... View more

You are getting a 550 SMTP error from your mail system (not Splunk). I would check the mail system you are pointed at when you try to send the message to determine the issue. Based on the message, the sending address may not be permitted to relay or to send to that recipient. It could also mean the address is not valid. Again, I would check the logs on the mail host at the time the message was sent to better troubleshoot the issue. ... View more

No new info just yet. There is a bug and it's being investigated. If you want to suppress the message, edit $SPLUNK_HOME/etc/log-cmdline.cfg, locate the following line: category.IniFile=INFO and change it to ERROR: category.IniFile=ERROR ... View more

I am not positive I understand the question but I'll give it a whirl. It sounds like you configured a sourcetype and the time stamping is set to auto. You want to understand what auto does or what it is using? It's not a specific settings which can be seen via btool. It's applying an "under the hood" logic to determine the time stamp. If you do not wish to use "auto" but instead have it configured, you could go through a similar process of importing the data in the gui, and this time when configuring the time stamping, select "Advanced" instead of auto. The following document has guidance on the configuration options and provide several examples: Configure Time Stamp Recognition ... View more