Hi, we are running a distributed Splunk environment and do monitor the messages which appearing when there are issues within the ecosystem.  We did read about how to customize messages and official Splunk docs for messages.conf but weren't able to receive good answers to that. Maybe one of you does have more experience with that https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Customizeuserexperience https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Messagesconf Can someone help to explain those parameters and the behavior? target = [auto|ui|log|ui,log|none] * Sets the message display target. * "auto" means the message display target is automatically determined by context. * "ui" messages are displayed in Splunk Web and can be passed on from search peers to search heads in a distributed search environment. * "log" messages are displayed only in the log files for the instance under the BulletinBoard component, with log levels that respect their message severity. For example, messages with severity "info" are displayed as INFO log entries. * "ui,log" combines the functions of the "ui" and "log" options. * "none" completely hides the message. (Please consider using "log" and reducing severity instead. Using "none" might impact diagnosability.) * Default: auto I try to find a way to control if messages are getting distributed to another instance like Monitoring Console or if they should only appear on the system where the issue  happend. Is that possible? Where do I find those event if I select "log" as parameter? do they appear only in splunkd.log? Thanks     ... View more

Hello together, we have a 3 node SH-Cluster where one member is not getting up again. If we want to restart the Splunk daemon it will stuck on the very last task to start the web server. After a while we are getting a WARNING: web interface does not seem to be available! On the newly selected captain node I've checked the kv status for the specific host: configVersion : -1 hostAndPort : <ip>:8191 lastHeartbeat : Mon Apr 15 .... lastHeartbeatRecv : ZERO_TIME lastHeartbeatRecvSec: 0 . . . replicationStatus : Down uptime : 0 When I search for error logs in the _internal logs I can see following messages in mongod logs: REPL [ReplicationExecutor] Error in heartbeat request to <own-ip-address>.8191; HostUnreachable: Connection refused ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to <own-ip-address>:8191 - HostUnreachable: Connection refused Should this ip address be the address of the captain? splunkd logs doesn't indicate any errors. For me it seems like the syncronisation of the kv store doesn't work. I've tried this already, but it didn't help: https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/ResyncKVstore any suggestions? Thanks! ... View more

Hello, to let you understand the problem easier I'll explain quickly the requirements for my role: - create a role which only have access to one specific dashboard which contains about 5 searches. - The role has no permissions to any index. - users of the roles need to be able to change their own password. we have created a splunk role with almost no capabillities (change_own_password, list_inputs, rest_properties_get, run_collect, run_mcollect, search). The dashboard is configured within an app which grants the user read permissions, but the user is not allowed to search against the indexs. All of this stuff is already working. To keep the ability to see reports without having permissois to execute searches on the corresponding indexes, we configured on the report settigs that it is getting executed by another user. The problem we have is that if you want to keep the ability to change its own password you need to grant the user "read" permissions for the search&reporting app. But this will also enable to navigate to "activity" -> "jobs" which gives the user the permissions to see scheduled searches and so on. When he clicks on one of those searches the user is able to see the search querys. But we want to avoid that the user is able to see the search query. Anyone know a best way to solve this? ... View more

Hello together, today we tried to fix the line breaking and timestamp problems we have with logs from cyberark.: <5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-TEST_MAIL_DualControl.ini dvc= shost=192.169.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_InvestigationCenter-TEST.ini dvc= shost=192.168.122.222 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-BLUB_InvestigationCenter.ini dvc= shost=192.168.122.221 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_GTX_Bottomline.ini dvc= shost=192.168.178.212 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg=<5>1 2019-02-02T10:14:37Z hostid CEF:0|Cyber-Ark|Vault|10.5.0000|51|Retrieve File|5|act=Retrieve File suser=PasswordManager fname=Root\Policies\Policy-SWIFT_GTX_Bottomline-TEST.ini dvc= shost=192.122.122.122 dhost= duser= externalId= app= reason= cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=PasswordManagerShared cs3Label="Device Type" cs3= cs4Label="Database" cs4= cs5Label="Other info" cs5= cn1Label="Request Id" cn1= cn2Label="Ticket Id" cn2= msg= Timestamp problem fixed with following configuration: MAX_TIMESTAMP_LOOKAHEAD = 30 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %Y-%m-%dT%H:%M:%S TIME_PREFIX = \<\d\>\d But for any reason I can't change Line Breaks for this text. Actually I want to break the event everytime when one of those <5>1 appears. Does anyone has an idea why? ... View more

Hello, Just a simple question about pretrained sourcetypes like 'syslog' and 'linux_syslog_messages'. Why are those props and transforms preconfigured to sanitize the hostname? For example, when I have a system with the hostname: "blub.local.com" it will strip this hostname to "blub". Actually I've heard that Splunk doesn't suggest to change the hostname in a props/transforms if it isn't necessary . You can still change it in your inputs which doesn't affect your performance that much. Furthermore when you want to 'splunk' some other logs they will have your real name "blub.local.com". I can change it manually from /opt/splunk/etc/system/default/ but that is actually just a workaround until the next update. Does anyone have an idea how to handle this very easily? Thanks and regards ... View more