About ws

ws

Hi, After setting up a test index and ingesting a test record, I’m now planning to remove the index from the distributed setup. Could anyone confirm the correct procedure for removing an index in a distributed environment with 3 indexers and a management node? I normally run the following command at an all in one setup. /opt/splunk/bin/splunk clean eventdata -index index_name

ws

Update I suspect this might be a Splunk-related issue, possibly due to the version I'm currently using (9.3.1). I spun up a new server for quick testing and reused the same configuration parameters from my previous setup. Mainly, the props.conf, transforms.conf, and inputs.conf. Interestingly, everything seems to be working fine on the new server, even though the configuration is identical to the old one. The only difference I can observe is in the data ingestion flow. initially, I ingested a set of JSON array entries in one format, and later ingested another set with a different structure containing more fields. So far, it all appears to be working as expected. However, when I tried the same method on my previous server, it didn’t work as expected. This is puzzling since both servers are using the same configuration files and setup. The only noticeable difference was the data ingestion flow on the new server, I ingested one format of JSON array first, followed by another with more fields, and it worked fine. But replicating this exact process on the older server doesn’t yield the same results.

ws

Hi @livehybrid, I'm still under to get the fields listed even updating the props.conf. [preprocess_case] TRANSFORMS-setsourcetype = sourcetype_router, sourcetype_router2 SHOULD_LINEMERGE=false LINE_BREAKER=(\[)|(([\r\n]+)\s*{(?=\s*"attribute":\s*{))|(\]) TRUNCATE=100000 TIME_PREFIX="ClosedDate":\s*" [too_small] PREFIX_SOURCETYPE = false

ws

Hi @livehybrid, i tried the following method to write into the local file with keeping the file at /tmp but it still didn't work. As for my situation, i think the best scenario would be keep a record of something like "seen before record.txt" and do a comparison and only to write new records into the file and remove previous indexed entries. At least the current approach is workable, but we’ll need to monitor the file size of "seen before record.txt" as it continues to grow. For now, the file size isn’t a concern since it only stores a limited number of tracking records.

ws

Hi, Unsure what is the root cause as i was trying to do some minor adjustment to ignore the [ ] at the transforms.conf. Previously I'm able to view the fields like Id Name and their value but currently nothing shows. I tried to re-do the props.conf, transforms.conf and inputs.conf by adding parameter by parameter and it still didn't work.

ws

@livehybrid, ok let me test out the following method as mention to download the file as a temp file, then write the contents to the existing file. I believe this can be handled within the same Python script, which connects to the FTP server and downloads the file to my local Splunk server. Thanks for sharing the additional information. Since I'm still learning, could you advise which log file I should be checking after enabling DEBUG for the following? Change the logging to DEBUG for the following components: TailingProcessor BatchReader WatchedFile FileTracker

ws

My original Python script accessed the FTP server directly and used the mget command to retrieve files from the monitored folder. But as mentioned by you, to try pulling the file from the FTP server into my local Splunk server into a different directory, before copying it on the splunk server to the monitored directory. I did a slight chance to the script to only cp after it exit from the FTP server. ftp -inv "$HOST" <<EOF >> /home/ws/fetch_debug.log 2>&1 user $USER $PASS cd $REMOTE_DIR lcd /home/ws/pull mget * bye EOF cp -v /home/ws/pull/*.json /home/ws/logs >> /home/ws/fetch_debug.log 2>&1

ws

Yes, I'm accessing my FTP server using the FTP method. However, this shouldn't make a difference whether I'm using FTP or SFTP, right? I'm still encountering the same issue, even after copying the file to a different folder before moving it to the monitored directory on the Splunk server. Just to add on, my file type is JSON. [Mon Apr 21 20:28:01 +08 2025] Attempting FTP to 192.168.80.139 Connected to 192.168.80.139 (192.168.80.139). 220 (vsFTPd 3.0.3) 331 Please specify the password. 230 Login successful. 250 Directory successfully changed. Local directory now /home/ws/pull 221 Goodbye. '/home/ws/pull/###_case_final.json' -> '/home/ws/logs/###_case_final.json' [Mon Apr 21 20:28:12 +08 2025] Attempting FTP to 192.168.80.139 Connected to 192.168.80.139 (192.168.80.139). 220 (vsFTPd 3.0.3) 331 Please specify the password. 230 Login successful. 250 Directory successfully changed. Local directory now /home/ws/pull local: ###_case_final.json remote: ###_case_final.json 227 Entering Passive Mode (192,168,80,139,249,175). 150 Opening BINARY mode data connection for ###_case_final.json (1455 bytes). 226 Transfer complete. 1455 bytes received in 8.5e-05 secs (17117.65 Kbytes/sec) 221 Goodbye. '/home/ws/pull/###_case_final.json' -> '/home/ws/logs/###_case_final.json' As of now, my inputs.conf contain the following only.

ws

Is there any solution to what i'm facing?? Here's what I’ve tested so far. 1: WinSCP uploads file.json to the FTP server → Splunk local server retrieves the file to a local directory → Splunk reads and indexes the data. sha256sum /splunk_local/file.json 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd 2: Deleted file.json from the FTP server → Used WinSCP to re-upload the same file.json → Splunk local server pulled the file to the local directory → Splunk did not index the file.json sha256sum /splunk_local/file.json 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd 3: WinSCP overwrote file.json on the FTP server with a version containing both new and existing entries → Splunk local server pulled the updated file to the local directory → Splunk re-read and re-indexed the entire file, including previously indexed data sha256sum /splunk_local/file.json 2217ee097b7d77ed4b2eabc695b89e5f30d4e8b85c8cbd261613ce65cda0b851 I noticed that the SHA value only changes when a new entry is added to the file, as seen in scenario 3. However, in scenarios 1 and 2, the SHA value remains the same—even if I delete and re-upload the exact same file to the FTP server and pull it into my local Splunk server. And yes, I'm pulling the file from the FTP server into my local Splunk server, where the file is being monitored.

ws

Here's what I’ve tested so far. 1: WinSCP uploads file.json to the FTP server → Splunk local server retrieves the file to a local directory → Splunk reads and indexes the data. sha256sum /splunk_local/file.json 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd 2: Deleted file.json from the FTP server → Used WinSCP to re-upload the same file.json → Splunk local server pulled the file to the local directory → Splunk did not index the file.json sha256sum /splunk_local/file.json 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd 3: WinSCP overwrote file.json on the FTP server with a version containing both new and existing entries → Splunk local server pulled the updated file to the local directory → Splunk re-read and re-indexed the entire file, including previously indexed data sha256sum /splunk_local/file.json 2217ee097b7d77ed4b2eabc695b89e5f30d4e8b85c8cbd261613ce65cda0b851 I noticed that the SHA value only changes when a new entry is added to the file, as seen in scenario 3. However, in scenarios 1 and 2, the SHA value remains the same—even if I delete and re-upload the exact same file to the FTP server and pull it into my local Splunk server. And yes, I'm pulling the file from the FTP server into my local Splunk server, where the file is being monitored.

ws

Hi, I'm facing an issue where the same data gets indexed multiple times every time the JSON file is pulled from the FTP server. Each time the JSON file is retrieved and placed on my local Splunk server, it overwrites the existing file. I don't have control over the content being placed on the FTP server, it could either be an entirely new entry or an existing entry with new data added, as shown below. I'm monitoring a specific file, as its name, type, and path remain consistent. From what I can observe, every time the file has new entries alongside previously indexed data, it is re-indexed, causing duplication. Example: file.json 2024-04-21 14:00 - row 1 2024-04-21 14:10 - row 2 overwritten file.json 2024-04-21 14:00 - row 1 2024-04-21 14:10 - row 2 2024-04-21 14:20 - row 3 Additionally, I checked the sha256sum of the JSON file after it’s pulled into my local Splunk server. The hash value changes before and after the file is overwritten. file.json: 2217ee097b7d77ed4b2eabc695b89e5f30d4e8b85c8cbd261613ce65cda0b851 /home/ws/logs/###.json overwritten file.json: 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd /home/ws/logs//###.json I've tried using initCrcLength, crcSalt, and followTail, but they don't seem to prevent the duplication, as Splunk still indexes it as new data. Any assistance would be appreciated, as I can't seem to prevent the duplication in indexing.

ws

Hi @livehybrid, Previously, I was monitoring a folder path. However, after making some adjustments, I’ve switched to monitoring a specific file instead, since the name, type, and path will always remain consistent. Now, I'm encountering an issue where the same data gets indexed multiple times whenever the JSON file is pulled from the FTP server. Each time the JSON file is retrieved and placed on my local Splunk server, it overwrites the existing file. I’ve tried using initCrcLength and crcSalt, but they don’t seem to prevent the duplication, as Splunk still indexes it as new data. Additionally, I checked the sha256sum of the JSON file after it’s pulled into my local Splunk server. The hash value changes before and after the new data overwrites the file. I'm not entirely sure how Splunk determines the file’s initial 256-byte hash for comparison. 1: 2217ee097b7d77ed4b2eabc695b89e5f30d4e8b85c8cbd261613ce65cda0b851 /home/ws/logs/cpf_case_final.json 2: 45b01fabce6f2a75742c192143055d33e5aa28be3d2c3ad324dd2e0af5adf8dd /home/ws/logs/cpf_case_final.json

ws

Hi @kiran_panchavat, In my case, if I don't use INDEXED_EXTRACTIONS = JSON. Which I believe helps to automatically handle and ignore square brackets [] based on the detection of the JSON format. Since I'm using transforms.conf to assign a sourcetype, every time the file is ingested, the indexer treats the [ character as a separate event. Do you know if there's anyway to ignore the square brackets if i do not use INDEXED_EXTRACTIONS = JSON? Additionally, I've noticed another issue: whenever the JSON file gets overwritten with new content. Whether it contains previously indexed data or new data. My script pulls it again, and the indexer re-indexes the file, resulting in duplicate entries in the index.

ws

@livehybrid just to add on, after getting the data in. If the monitored folder json file has been update with either an appending of records or so. Do you happen to know why it indexes the whole json file again? rather than just the appended new records?? since the naming of the json remain the same.

ws

@kiran_panchavat, After several attempts in my situation, I tried using the following settings for JSON. While it was able to read the data, each record/value ended up having duplicated values. I tried setting the relevant KV options, but it still didn’t resolve the issue. For now, I’ve decided to proceed without using INDEXED_EXTRACTIONS. It still works, but it treats the [ as a single entry. I'm still unsure how to fully resolve this. *Just a heads up. I'm also using transforms.conf, though I'm not entirely sure if that's what's causing the duplicate values* INDEXED_EXTRACTIONS = JSON either with or without the following: KV_MODE = none AUTO_KV_JSON = false @livehybrid , Great! What you mentioned was part of the reason why two entries kept getting indexed together. After updating the configuration and removing the other stanza, I was able to index the JSON array as multiple events. I also noticed that it might have been due to my use of transforms.conf to assign the sourcetype.

ws

Hi @livehybrid, For testing purpose, my architecture is all-in-one setup. For my actual deployment, to mine understand the props.conf and transforms.conf will be at my HF right? since the pulling of the json file land at my HF local server.

ws

Hi @kiran_panchavat, I noticed that the sample provided aligns with what I’m trying to achieve. However, after applying the same settings for testing, I’m still not getting the same results as you. I’ve attached a screenshot for your reference—please help point out any mistakes or adjustments that may be needed. I don’t believe the issue lies with the transforms.conf configuration. JSON file: [ { "attribute":{ "type": "case" }, "Id": "I0000005", "name": "ws", "email": "ws@gmail.com", "case type__c": "Service Case", "date": "17/4/2025", "time": "16:15", "account":{ "attribute": { "type": "account" }, "Id": "I0000005" } }, { "attribute":{ "type": "case" }, "Id": "I0000006", "name": "thomas", "email": "thomas@gmail.com", "case type__c": "Transaction Case", "date": "17/4/2025", "time": "16:15", "account":{ "attribute": { "type": "account" }, "Id": "I0000006" } } ] Search Head: Props.conf Transforms.conf

ws

I'm looking for a way to split a JSON array into multiple events, but it keeps getting indexed as a single event. I've tried using various parameters in props.conf, but none of them seem to work. Does anyone know how to split the array into separate events based on my condition? I want it to appear as two sets of events. JSON string: Splunk Search Head:

ws · ‎02-18-2025

Anyone happen to know the following message? When i trigger a customize application, i get the follow message ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c1106)

ws · ‎02-13-2025

May i know where part should i refer to for the following error? ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c1106)

ws · ‎02-10-2025

My apologies if my explanation is confusing. You are right, the csr has been signed, so right now it's a certificate which is in .pem format. But the rather, the root ca certificate is in .cer format. And for my testing environment, the root ca certificate is in .pem format. My next step is trying to convert it but unsure will it work.

ws · ‎02-10-2025

Hi, I’m currently encountering the following error message in `splunkd.log` when I enable the custom TA Add-on. I have a Python script that successfully tests the signed CSR, private key, and root CA. It can establish a connection and retrieve logs as expected. However, when using the application created, I am seeing the error message. I’ve double-checked the values, and everything seems to be the same. In our testing environment, it works, but the only difference I noticed is that the root CA certificate is in .csr format. Should I convert it to .pem, as we did in the testing environment? -0700 ERROR ExecProcessor - message from "/data/splunk/bin/python3.7 /data/splunk/etc/apps/TA_case/bin/case.py" HTTPSConnectionPool(host='<HiddenForSensitivityPurpose>', port=443): Max retries exceeded with url: <HiddenForSensitivityPurpose>caseType=Service+Case&fromData=2025-02-06+17%3A23&endDate=2025-02-06+21%3A23 (Caused by SSLError(SSLCertverificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')))

ws · ‎02-09-2025

Understand Splunk will perform a check of the event at 256 chars if they are the same. But at my current situation, would your recommendation be that we need to customize the application to implement a checkpoint mechanism for tracking previously indexed records?

ws · ‎02-09-2025

Currently, we are not focusing on searches but rather on the application created to pull data from the API provided by the destination party. Based on my understanding of the current setup, the new data is being retrieved by the application through the destination API. The data includes fields such as ID, case status, case close date, and others. At this point, duplicates will be identified based on the ID field. Please correct me if I'm wrong, but given the current setup, wouldn't this result in duplicate data? Since we are calling at the interval of 1 hours and 4 hours duration of logs. For example: 10am, 6am-10am 11am, 11am-3pm

ws · ‎02-06-2025

Currently using an customize App to connect to a case / monitoring system and retrieve data. I found out that, Splunk has the ability to detect if the data has already been indexed. But if I have the following scenario? will it consider as a duplicate or new data? since it has a new close case timing for the update close case. One of the previously closed cases has been reopened and closed again with a new case closed time. will Splunk enterprise consider as a new data to index?

Posts	25
Solutions	0
Karma Given	6
Karma Received	1
Member Since	‎02-06-2025

Online Status	Offline
Date Last Visited	a week ago

Remove the index distributed setup

Unable to view value from events

How to avoid indexing events twice

How to split a json array into multiple events?

Splunk SSL Error with Python

How do we define a duplicate record?

Remove the index distributed setup

Re: Unable to view value from events

Re: Unable to view value from events

Re: How to avoid indexing events twice

Unable to view value from events

Re: How to avoid indexing events twice

Re: How to avoid indexing events twice

Re: How to avoid indexing events twice

Re: How to split a json array into multiple events...

Re: How to avoid indexing events twice

How to avoid indexing events twice

Re: How to split a json array into multiple events...

Re: How to split a json array into multiple events...

Re: How to split a json array into multiple events...

Re: How to split a json array into multiple events...

Re: How to split a json array into multiple events...

Re: How to split a json array into multiple events...

How to split a json array into multiple events?

Re: Splunk SSL Error with Python

Re: Splunk SSL Error with Python

Re: Splunk SSL Error with Python

Splunk SSL Error with Python

Re: How do we define a duplicate record?

Re: How do we define a duplicate record?

How do we define a duplicate record?

Are you a member of the Splunk Community?