Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- About dbray_sd
dbray_sd
Path Finder
Member since:
12-08-2015
Friday
Community Statistics
| Posts | 33 |
| Solutions | 5 |
| Karma Given | 4 |
| Karma Received | 6 |
| Member Since | 12-08-2015 |
Activity Feed
- Posted Re: How do I disable redirection warning? on Dashboards & Visualizations. Friday
- Posted Re: 9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found on All Apps and Add-ons. 01-17-2025 10:13 AM
- Posted Re: 9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found on All Apps and Add-ons. 01-15-2025 05:05 AM
- Posted Re: 9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found on All Apps and Add-ons. 01-10-2025 06:54 AM
- Posted Re: 9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found on All Apps and Add-ons. 01-10-2025 06:31 AM
- Posted 9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found on All Apps and Add-ons. 01-10-2025 04:23 AM
- Posted Re: Yet Another Duplicated JSON Field Data in Table on Getting Data In. 12-16-2024 06:47 AM
- Posted Re: Yet Another Duplicated JSON Field Data in Table on Getting Data In. 12-16-2024 04:40 AM
- Posted Re: Yet Another Duplicated JSON Field Data in Table on Getting Data In. 12-13-2024 09:52 AM
- Posted Re: Yet Another Duplicated JSON Field Data in Table on Getting Data In. 12-13-2024 08:06 AM
- Posted Yet Another Duplicated JSON Field Data in Table on Getting Data In. 12-13-2024 06:57 AM
- Got Karma for Re: ERROR: Failed to migrate to storage engine wiredTiger, reason=[blank]. 03-24-2022 09:03 AM
- Posted Re: ERROR: Failed to migrate to storage engine wiredTiger, reason=[blank] on Installation. 02-15-2022 08:32 AM
- Got Karma for Re: One Search Head ignores props.conf One Does Not. 08-18-2021 12:23 AM
- Got Karma for Re: One Search Head ignores props.conf One Does Not. 08-17-2021 05:37 AM
- Posted Re: One Search Head ignores props.conf One Does Not on Getting Data In. 08-17-2021 04:18 AM
- Posted Re: One Search Head ignores props.conf One Does Not on Getting Data In. 08-10-2021 04:35 AM
- Posted Re: One Search Head ignores props.conf One Does Not on Getting Data In. 08-09-2021 10:01 AM
- Posted Re: One Search Head ignores props.conf One Does Not on Getting Data In. 08-09-2021 09:56 AM
- Posted Re: One Search Head ignores props.conf One Does Not on Getting Data In. 08-09-2021 09:52 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
Friday
Why is this marked as solved, when it is not solved??? Over 3 years later, running version 9.4.x and still no way for Enterprise to stop this ridiculous behavior. Why Splunk, why??? Unless I'm missing something, but Settings -> Servers settings gives no solution to stop this.
... View more
01-17-2025
10:13 AM
Splunk support came back and stated this is a known issue, and the 9.4.0 update has an issue with the Splunk DB Connect app. The work around was time consuming, but finally everything is back up and running. I had to manually go into: /opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf ...and comment out each line with: tail_rising_column_init_ckpt_value checkpoint_key Then restart Splunk, then go into each INPUT config and manually reset the checkpoint value to what was recorded in the tail_rising_column_init_ckpt_value setting. Took forever, but after doing all that and another Splunk restart, only then did all the issues go away. Also noted that the 9.4.0 update removes the legacy tail_rising_column_init_ckpt_value from the db_inputs.conf file, as it is now stored in kvstore, and since kvstore has been updated with 9.4.0 update, that was the overall issue. Just yet another mess that Splunk updates have caused, but at least support is aware, and they are working hard to properly fix it.
... View more
01-15-2025
05:05 AM
Yes, and the issues follows over to the cloned entry.
... View more
01-10-2025
06:54 AM
Thank you, nothing useful in the logs, and I've already opened a ticket. I'll report back.
... View more
01-10-2025
06:31 AM
Thank you @kiran_panchavat that at least gives me something to investigate further, but also confusing. Health Check is complaining about: One or more defined connections require the corresponding JDBC driver. However, those JDBC drivers comes from the Splunk_JDBC_mysql add on app, which I checked and it's running with the latest version. Confusing.
... View more
01-10-2025
04:23 AM
After upgrading Splunk to 9.4.0 and Splunk DB Connect to 3.18.1, all INPUTS have the error: Checkpoint not found. The input in rising mode is expected to contain a checkpoint. None of them are pulling in data. Looking over the logs, I see: 2025-01-10 12:16:00.298 +0000 Trace-Id=1d3654ac-86c1-445f-97c6-6919b3f6eb8c [Scheduled-Job-Executor-116] ERROR org.easybatch.core.job.BatchJob - Unable to open record reader com.splunk.dbx.server.exception.ReadCheckpointFailException: Error(s) occur when reading checkpoint. at com.splunk.dbx.server.dbinput.task.DbInputCheckpointManager.load(DbInputCheckpointManager.java:71) at com.splunk.dbx.server.dbinput.task.DbInputTask.loadCheckpoint(DbInputTask.java:133) at com.splunk.dbx.server.dbinput.recordreader.DbInputRecordReader.executeQuery(DbInputRecordReader.java:82) at com.splunk.dbx.server.dbinput.recordreader.DbInputRecordReader.open(DbInputRecordReader.java:55) at org.easybatch.core.job.BatchJob.openReader(BatchJob.java:140) at org.easybatch.core.job.BatchJob.call(BatchJob.java:97) at com.splunk.dbx.server.api.service.conf.impl.InputServiceImpl.runTask(InputServiceImpl.java:321) at com.splunk.dbx.server.api.resource.InputResource.lambda$runInput$1(InputResource.java:183) at com.splunk.dbx.logging.MdcTaskDecorator.run(MdcTaskDecorator.java:23) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) I'm unable to Edit the config, and update the Check point value. Even thought the Execute Query works, when I try to save the update it gives: Error(s) occur when reading checkpoint. Has anybody else successfully upgraded to 9.4.0 and 3.18.1?
... View more
Labels
- Labels:
-
upgrade
12-16-2024
06:47 AM
Well, right as I go to create a ticket, I stumbled onto an old note that had the fix. I had to do the following on the SH: sudo -u splunk mkdir -p /opt/splunk/etc/apps/RabbitMQ_Settings/metadata sudo -u splunk vim /opt/splunk/etc/apps/RabbitMQ_Settings/metadata/local.meta []
access = read : [ * ], write : [ admin ]
export = system I recalled something about permissions or some other weirdo Splunk requirement. Aggravating, but at least it's working as expected now.
... View more
12-16-2024
04:40 AM
Well, unfortunately, as I stated above "neither fixes the issue." It doesn't matter how I configure the UF props.conf or the SH props.conf, Splunk is refusing to parse JSON properly. Even though other JSON datafeeds work just fine. Guess I'll have to open another ticket with Splunk.
... View more
12-13-2024
09:52 AM
Prefer the SH side to what? Have or not have INDEXED_EXTRACTIONS? Depends on what? The needs are simple. We want Splunk to not show duplicated fields when showing JSON data into tables., and be consistent. What is really confusing (and aggravating) about this, is we have other JSON feeds coming in, and those are working just fine. However, each config appears to be different. Sometimes the UF has the props.conf. Sometimes the SH has the props.conf. Sometimes the props.conf has "INDEXED_EXTRACTIONS = JSON" sometimes it doesn't. The whole thing is really confusing as to why Splunk sometimes works properly, and sometimes doesn't. The JSON file is rather large, so put it on pastebin: https://pastebin.com/VwkcdLLA Appreciate your attention with this confusing issue. Let me know if you have any other questions.
... View more
12-13-2024
08:06 AM
From which one? The Search Head or the Universal Forwarder.....because I can already tell you, neither fixes the issue. Removed from Search Head, issue remains. Put it back on the Search Head, removed from the Universal Forwarder, issue changes into it not parsing the data as JSON, so it loses all formatting, and the table is blank.
... View more
12-13-2024
06:57 AM
I have seen a lot of similar Questions/Solutions with this aggravating issue, none of which are working. Trying to pull RabbitMQ API (JSON) data into Splunk. Bash script creates /opt/data/rabbitmq-queues.json curl -s -u test:test https://localhost:15671/api/queues | jq > /opt/data/rabbitmq-queues.json Universal Forwarder has following props.conf on the RabbitMQ server: [rabbitmq:queues:json]
AUTO_KV_JSON = false
INDEXED_EXTRACTIONS = JSON
KV_MODE = none And the inputs.conf: [batch:///opt/data/rabbitmq-queues.json]
disabled = false
index = rabbitmq
sourcetype = rabbitmq:queues:json
move_policy = sinkhole
crcSalt = <SOURCE>
initCrcLength = 1048576 We run the btool on the Universal Forwarder to verify the settings are getting applied correctly: sudo /opt/splunkforwarder/bin/splunk btool props list --debug "rabbitmq:queues:json"
/opt/splunkforwarder/etc/apps/RabbitMQ_Settings/local/props.conf [rabbitmq:queues:json]
/opt/splunkforwarder/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunkforwarder/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunkforwarder/etc/apps/RabbitMQ_Settings/local/props.conf AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunkforwarder/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunkforwarder/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunkforwarder/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunkforwarder/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunkforwarder/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunkforwarder/etc/system/default/props.conf HEADER_MODE =
/opt/splunkforwarder/etc/apps/RabbitMQ_Settings/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/apps/RabbitMQ_Settings/local/props.conf KV_MODE = none
/opt/splunkforwarder/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunkforwarder/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunkforwarder/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunkforwarder/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunkforwarder/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunkforwarder/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunkforwarder/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunkforwarder/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunkforwarder/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunkforwarder/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunkforwarder/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunkforwarder/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunkforwarder/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunkforwarder/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunkforwarder/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunkforwarder/etc/system/default/props.conf SHOULD_LINEMERGE = True
/opt/splunkforwarder/etc/system/default/props.conf TRANSFORMS =
/opt/splunkforwarder/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunkforwarder/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunkforwarder/etc/system/default/props.conf maxDist = 100
/opt/splunkforwarder/etc/system/default/props.conf priority =
/opt/splunkforwarder/etc/system/default/props.conf sourcetype =
/opt/splunkforwarder/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunkforwarder/etc/system/default/props.conf unarchive_cmd_start_mode = shell On the local Search Head we have the following props.conf: [rabbitmq:queues:json]
KV_MODE = none
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false We run the btool on the Search Head to verify the settings are getting applied correctly: sudo -u splunk /opt/splunk/bin/splunk btool props list --debug "rabbitmq:queues:json"
/opt/splunk/etc/apps/RabbitMQ_Settings/local/props.conf [rabbitmq:queues:json]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/apps/RabbitMQ_Settings/local/props.conf AUTO_KV_JSON = false
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/apps/RabbitMQ_Settings/local/props.conf INDEXED_EXTRACTIONS = json
/opt/splunk/etc/apps/RabbitMQ_Settings/local/props.conf KV_MODE = none
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/system/default/props.conf unarchive_cmd_start_mode = shell However, even with all that in place, we're still seeing duplicate values when using tables: index="rabbitmq"
| table _time messages state
... View more
Labels
- Labels:
-
JSON
02-15-2022
08:32 AM
1 Karma
Same issue, same blank result, and same sad and unhelpful log files. Pretty much business as usual for Splunk. Not even sure why I'm surprised anymore. Miss the good ol' days when Splunk was worth the hefty price tag.
... View more
08-17-2021
04:18 AM
2 Karma
Unfortunately, I had to give up on my investigation, and ended up re-installing the broke SH. I removed Splunk, reinstalled the same rpm, reinstalled the same troublesome app, and everything worked (as expected). Then, I moved one app back in at a time, each time checking the original broke props.conf/app and each time everything continued to work. So, no real good idea what was causing the issue, but everything is working now.
... View more
08-10-2021
04:35 AM
Not really sure I understand exactly what you are asking to confirm. The app doesn't matter. I can duplicate this issue in our customized app, in the Search & reporting app, or any other place. The particular app that contains the specific props.conf file is set with Global permissions. In the new working SH, it also doesn't matter where or how I perform the searches. It always uses the Global permissions on the props.conf file, and always works.
... View more
08-09-2021
10:01 AM
Yes, same roles same, LDAP settings, same everything (as far as I can tell). No search filters defined.
... View more
08-09-2021
09:56 AM
Good theory, but no, that's not it. All users are having the issue on the old server, but not the new.
... View more
08-09-2021
09:52 AM
Sorry, yes of course. I've even gone as far as updating the entire environment to the latest Splunk version, to make sure it was not some older bug. So now, everything is upgraded to the latest released version, and restarted.
... View more
08-09-2021
05:49 AM
We have 3 clustered indexers and an original Search Head. Installed an app that has a custom props.conf on the Search Head, and it is NOT showing the extracted proper fields when performing searches. Deployed a new Search Head and installed the same exact app. The new Search Head shows the proper fields. The two servers appear to be identical, and running: splunk cmd btool props list --debug ...shows the same exact results, line by line, for the app. The original server does have some extra apps, but with the results of the btool above, it would not appear there are any conflicts with other apps. What would be the next steps in troubleshooting the original Search Head, and why it does not show the proper fields?
... View more
Labels
- Labels:
-
props.conf
05-21-2021
04:16 AM
Wow, what a simple typo that was really hard to see until I took the time and ran: sudo -u splunk /opt/splunk/bin/splunk list inputstatus /opt/splunk/var/run/splunk/csv/test.csv
parent = opt/splunk/var/run/splunk/csv/*.csv
type = File did not match whitelist '^opt\/splunk\/var\/run\/splunk\/csv/[^/]*\.csv$'. That little ^opt at the beginning showed me that I was missing an extra "/" in: [batch://opt/splunk/var/run/splunk/csv/*.csv] It should be: [batch:///opt/splunk/var/run/splunk/csv/*.csv] So, all good to go.
... View more
05-14-2021
08:22 AM
Here is the inputs.conf entry: [batch://opt/splunk/var/run/splunk/csv/*.csv]
disabled = false
move_policy = sinkhole
index = test-metrics
sourcetype = metrics_csv However, as I monitor /opt/splunk/var/run/splunk/csv/ I see the CSV files are still there, and not getting indexed. This should have been a really simple test, but can't figure out why batch is not working. If I hardcode a specific CSV file it works: [batch://opt/splunk/var/run/splunk/csv/test.csv]
disabled = false
move_policy = sinkhole
index = test-metrics
sourcetype = metrics_csv But obviously I need it to get all the CSV files, so I should be able to use the wildcard *.csv
... View more
Labels
- Labels:
-
inputs.conf
02-18-2019
12:14 PM
Did you ever get this resolved? I seem to be having the same issue.
... View more
01-17-2018
10:33 AM
I've recently upgraded to Splunk 7.0.1 and palo-alto-networks-add-on-for-splunk_602 with palo-alto-networks-app-for-splunk_601. I have noticed that for the empty Dashboards within the app, if I crack them open and put:
index="pan_logs"
...in front of each query, then the data is populated. Or, easier, update the search macros to include the specific index and that works as well. However, without specifying the index, the data remains blank. I have even tried to revert back to palo-alto-networks-add-on-for-splunk_382 and palo-alto-networks-app-for-splunk_542, but the issue remains. Anybody else seeing this weirdness?
... View more
05-25-2017
06:58 AM
Could you explain a little further? Maybe some examples? I think this is the exact problem I'm having, but not really understanding what Splunk wants for this simple dropdown.
... View more
08-10-2016
12:36 PM
Thank you. The above mixed with the some extra evals will give exactly what I needed. A true uptime (donwtime) of the device. Here is the complete end result:
status_code=DOWN OR status_code=UP host_name="host1" |
addinfo |
streamstats range(_time) as delta count reset_on_change=true by status_code |
where count=1 |
reverse |
delta _time as Duration |
where status_code="DOWN" |
eval TotalTime=(info_max_time - info_min_time) |
eval PercDown=round((Duration / TotalTime * 100),3) |
eval PercUp=(100.000 - PercDown) |
table host_name PercUp PercDown
There might be a better way, and there is probably some more tweaks to make this work with multiple hosts. But at least I'm one step closer.
... View more
