Hello, first of all, sorry for my bad English, I hope you can understand everything. My goal is to get the journald logs from the universalforwarder in JSON format to Splunk. (Splunk/UF Version 9.1.2) I use the app jorunald_input. inputs.conf (UF)     [journald://sshd] index = test sourcetype = test journalctl-filter = _SYSTEMD_UNIT=sshd.service       I've tried different props.conf functions. For example, something like this:  props.conf (UF)     [test] INDEXED_EXTRACTIONS = json KV_MODE = json SHOULD_LINEMERGE=false #INDEXED_EXTRACTIONS =json #NO_BINARY_CHECK=true #AUTO_KV_JSON = true       On the UF I check with the command     ps aux | grep journalctl     whether the query is enabled. It displays this command     journalctl -f -o json --after-cursor s=a12345ab1abc12ab12345a01f1e920538;i=43a2c;b=c7efb124c33f43b0b0142ca0901ca8de;m=11aa0e450a21;t=233ae3422cd31;x=00af2c733a2cdfe7 _SYSTEMD_UNIT=sshd.service -q --output-fields PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE,MESSAGE     I can try it out by using this command in the cli But I have to take out that part "--after-cursor ...." So I run the following command on the CLI to keep track of the journald logs:     journalctl -f -o json _SYSTEMD_UNIT=sshd.service -q --output-fields PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE,MESSAGE     On the Universal forwarder, the tracked journald logs will then look like this:  (It would be a nice JSON format)     { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2c;b=a1aaa111a11aaa111aa000a0101;m=11aa00c5b9a0;t=233ae39a37aa2;x=00af2c733a2cdfe7", "__REALTIME_TIMESTAMP" : "1710831664593570", "__MONOTONIC_TIMESTAMP" : "27194940570016", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "Invalid user asdf from 111.11.111.111 port 111", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2d;b=a1aaa111a11aaa111aa000a0101;m=11aa00ec25bf;t=233ae39c9e6c0;x=10ac2c735c2cdfe7", "__REALTIME_TIMESTAMP" : "1710831667111616", "__MONOTONIC_TIMESTAMP" : "27194943088063", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): check pass; user unknown", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2e;b=a1aaa111a11aaa111aa000a0101;m=11aa00ec278a;t=233ae39c9e88c;x=5fb4c21ae6130519", "__REALTIME_TIMESTAMP" : "1710831667112076", "__MONOTONIC_TIMESTAMP" : "27194943088522", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2f;b=a1aaa111a11aaa111aa000a0101;m=11aa0108f5bf;t=233ae39e6b6c0;x=d072e90acf887129", "__REALTIME_TIMESTAMP" : "1710831668999872", "__MONOTONIC_TIMESTAMP" : "27194944976319", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a30;b=a1aaa111a11aaa111aa000a0101;m=11aa010e0295;t=233ae39ebc397;x=d1eb29e00003daa7", "__REALTIME_TIMESTAMP" : "1710831669330839", "__MONOTONIC_TIMESTAMP" : "27194945307285", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): check pass; user unknown", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a31;b=a1aaa111a11aaa111aa000a0101;m=11aa012f0b3c;t=233ae3a0ccc3e;x=c33e28a6111c89ea", "__REALTIME_TIMESTAMP" : "1710831671495742", "__MONOTONIC_TIMESTAMP" : "27194947472188", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a32;b=a1aaa111a11aaa111aa000a0101;m=11aa0135591b;t=233ae3a131a1d;x=45420f6d2ca07377", "__REALTIME_TIMESTAMP" : "1710831671908893", "__MONOTONIC_TIMESTAMP" : "27194947885339", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "PRIORITY" : "3", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "error: Received disconnect from 111.11.111.111 port 111:11: Unable to authenticate [preauth]" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a33;b=a1aaa111a11aaa111aa000a0101;m=11aa01355bee;t=233ae3a131cf0;x=15b1aa1201a45cdf", "__REALTIME_TIMESTAMP" : "1710831671909616", "__MONOTONIC_TIMESTAMP" : "27194947886062", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Disconnected from invalid user asdf 111.11.111.111 port 111 [preauth]" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a34;b=a1aaa111a11aaa111aa000a0101;m=11aa01355c42;t=233ae3a131d45;x=123f45a09e00a8a2", "__REALTIME_TIMESTAMP" : "1710831671909701", "__MONOTONIC_TIMESTAMP" : "27194947886146", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111", "_PID" : "1430615" }      (Example)    But when I look for the logs on the search head, they look like this:     Invalid user asdf from 111.11.111.111 port 111pam_unix(sshd:auth): check pass; user unknownpam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2pam_unix(sshd:auth): check pass; user unknownFailed password for invalid user asdf from 111.11.111.111 port 111 ssh2error: Received disconnect from 111.11.111.111 port 111:11: Unable to authenticate [preauth]Disconnected from invalid user asdf 111.11.111.111 port 111 [preauth]PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111       Does anyone know why the logs are written together and not to be considered individually? And why the logs are not in JSON format? Can anyone tell me a solution for this on how to fix the problem?   Thank you very much!

In our dashboard, a user reported that she got "Search was cancelled" message when she used it. I came to know that multiple searches were run concurrently. Hence, it could have been cancelled. But when I tried to reproduce this issue, the searches kept getting queued and ran but was never delayed.  What are the scenario when a search is queued vs when it is cancelled?

Hello, I have been working on Splunk for a few months now, and we are using Splunk mainly for Cyber Security monitoring. I am wondering with regards to data model (CIM) should I create separate data model for different zones, or should I combine all in a single data model. For example, I have 3 different independent network zones, DMZ, Zone A and Zone B. Each of those zones will have multiple indexes linked to it. Shall I actually use the default data model in CIM, eg datamodel=Authentication with all the indexes in DMZ, ZoneA and ZoneB, or should I make copies of datamodel? Scenario 1: If I use a common data model, I will use where source=xxx for example to try to split things out for my queries and dashboarding. Scenario 2: If I use a separate data model, I will have, datamodel= DMZ_Authentication, datamodel=ZoneA_Authentication and perhaps use append when I need to see the overall picture? Still confused on which is the best approach.

Hi Splunkers, I'm currently working on customizing the Splunk login screen to change the logo, background, footer, etc. I referred to the Splunk documentation (https://docs.splunk.com/Documentation/Splunk/9.1.3/AdvancedDev/CustomizeLogin) and successfully completed the customization. Now, the Splunk login screen displays my logo, background image, and footer content. However, I encountered an issue when running Splunk App Inspect on the custom-login app I created. The Splunk App Inspect tool reported 8 failures: The web.conf file contains a [settings] stanza, which is not permitted. Only [endpoint:] and [expose:] stanzas are allowed in web.conf. Line Number: 1 The 'local' directory exists, which is not allowed. All configuration should be in the 'default' directory. The 'local.meta' file exists, which is not permitted. All metadata permissions should be set in 'default.meta'. No README file was found, which should include version support, system requirements, installation, configuration, troubleshooting, and running of the app, or a link to online documentation. I'm wondering if the [endpoint:*] and [expose:*] stanzas in web.conf are necessary for customizing the login screen. Are these stanzas required for login screen changes? All other issues have been fixed. (for production environment) Below is the corrected version of the custom_login app structure based on the Splunk App Inspect recommendations: ``` custom_login |-- default | |-- app.conf | |-- web.conf | |-- restmap.conf | |-- savedsearches.conf | |-- ui_prefs.conf | |-- README | |-- data |-- metadata | |-- default.meta |-- static | |-- appIcon.png | |-- appIcon_2x.png | |-- appIconAlt.png | |-- appIconAlt_2x.png | |-- background.png | |-- fav_logo.png |-- bin | |-- readme.txt |-- appserver | |-- static | |-- background.png | |-- fav_logo.png ``` Here are the contents of the configuration files: **app.conf** ``` [launcher] author = NAME description = <<<<<XXXXXXXXYYYYYY>>>>>>. version = Custom login 1.0 [install] is_configured = 0 [ui] is_visible = 1 label = Custom app login [triggers] reload.web = web.conf reload.restmap = restmap.conf reload.ui_prefs = ui_prefs.conf ``` **restmap.conf** ``` # restmap.conf for custom_login [endpoint:login_background] # REST endpoint for login background image configurations match = /custom_login ``` **ui_prefs.conf** ``` [settings] loginBackgroundImageOption = custom loginCustomBackgroundImage = custom_login:appserver/static/background.png login_content = This is a server managed by my team. For any inquiries, please reach out to us at YYYY.com loginCustomLogo = custom_login:appserver/static/fav_logo.png loginFooterOption = custom loginFooterText = © 2024 XXXXXXX ``` **web.conf** ``` [endpoint:login] [expose:login_background] pattern = /custom_login methods = GET ``` I am currently working in a development environment. Any advice on how to proceed with these changes would be appreciated. Thanks in advance.