About tungpx

tungpx · ‎12-26-2024

Hello all, I want to ask about the mechanic of rolling bucket from hot to cold. In our indexes.conf we don't setup a warm path, just hot and cold control by maxDataSizeMB. System team give me 1TB of SSD and 3TB of SAS to work with. So naturally, I put hot path to the SSD and cold path to the SAS. Now we are encountering the problems that the indexingQueue always fill up to 100 whenever that indexer ingest data. So my question is: 1. Does the process of rolling bucket from hot to cold affect the IOPS and the writting in indexingQueue? 2. My understanding is that, the data flow go like this Forwarder -> indexer hot -> indexer cold, and this is a continuous process. And in case indexer hot is max out, it will roll to cold, but cold is SAS so the writing speed is < SSD. For example hot ingesting 2000 events per sec, but only push out 500 events per sec to cold, but hot is full already so it render the effective ingesting speed of hot to only 500 (since it full and can only take in the amount that it can push out). Is this correct? 3. If my understanding is correct, how should I approach in optimizing it. I'm thinking of two option: a) Switch our retention policy from size base to day base, setting hot retention to 1 day, cold remain size retention, since we ingested 600~800GB per day, we can ensure the hot partion will always have a buffer to ensure the smooth transition. My question in this section is when is the rolling happen, at the end of the day, or whenever the event is one day old, thus don't change anything. b) Create a warm path as a buffer, hot->warm->cold, the warm bucket will have 1TB and retention of 1 day, so, and with how we ingest 600-800GB per day, the warm path will always have space for the hot to roll over Is there anything else can I do?

tungpx · ‎12-25-2024

Yes, they're UFs. I already set [thruput] maxKBps = 0 in limits.conf in the app.

tungpx · ‎12-24-2024

After some investigation, the answer is: 1) The OS is Linux Redhat 8, Splunk UF version 9.1.1, we have 2 deployment of Splunk which is Splunk Enterprise and Splunk Security, on my end (Splunk Enterprise) there are only 2 inputs but on the Security end, there are a lot, with 2 apps HG_TA_Splunk_Nix and TA_nmon (roughly 40 inputs each) over 4 hosts. 2.1) There are some but not noteworthy ERROR. The errors are below: +700 ERROR TcpoutputQ [11073 TcpOutEloop] - Unexpected event id=<eventid> -> benign ERROR as per Splunk dev +700 ERROR ExecProcessor [32056 ExecProcessor] - message from "$SPLUNKHOME/HG_TA_Splunk_Nix/bin/update.sh" https://repo.napas.local/centos/7/updates/x84_64/repodata/repomd.xml: [Errorno14] curl#7 - "Failed to connect to repo.napas.local:80; No route to host" 2.2) HealthReporter show +700 INFO PeriodHealthReporter - feature="Ingestion latency" color=red/yellow indicator="ingestion_latency_gap_multiplier" due_to_threshold_value=1 measured_value=26684 reason=Events from tracker.log have not been seen for the last 26684 seconds, which is more than the red threshold ( 210 seconds ). This typically occurs when indexing or forwarding are falling behind or are blocked." node_type=indicator node_path=splunkd.file_monitor_input.ingestion_latency.ingestion_latency_gap_multiplier. 2.3) log _internal |stats count by destIP show idx1: 14248 idx2: 8014 idx3: 7963 idx4: 7809 Which is more concerning than I thought it would be. 2.4) Another find. The log is now lagging 1 hour behind, and still being pulled/ingest. But the internal log had stop, the time now is 9:08, but the last internal log is 8:19, with no error, which is +700 Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.000, instantaneous_eps=0.000, average_kbps=0.000, total_k_processed=0.000, kb=0.000, ev=0, interval_sec=60

tungpx · ‎12-24-2024

Hello, I have a case where the logs from 4 host are lagging behind. Why I say inconsistant is the laggig is differ from 5 to 30 minutes, sometime didn't at all. When the log don't show up 30 minutes or more, I go to the forwarder management and disable/enable apps, restart Splunkd, then the log continue with 1, 2 seconds lag. The other host also lagging behind at peak hour, but only for 1 or 2 minutes (maximum 5' for source with large amount of logs). I admit that our indexer cluster is not up to par in IOPS requirement but for 4 paticular host to be visible underperform is quite concerning. Can someone show me steps to debug and solve the problems.

tungpx · ‎10-21-2024

Hello, I have a deployment server and deploy an app on an Universal Forwarder, like I usually do (Create an app folder -> create local folder -> write input.conf -> setup app, server class on DS, tick disable/enable app, tick restart Splunkd). But after make sure the log path and permission of the log file (664), I don't see the log forwarded. I'm only manage the Splunk Deloyment but not the server that host universal forwarder so I asked the system team to check it for me. After sometime, they get back to me and said there is no change on the input.conf file. They have to manually restart splunk on the Universal Forwarder and after that I see the log finally ingested. So I want to know if there is an app, or a way to check if the app or the input.conf was changed according to my config on the DS or not, I can't ask the system team to check for it for me all time time. Thank you.

tungpx · ‎09-19-2024

I do have Splunk Enterprise license and my Splunk version is 9.1.1. The problem I have is anyone can access this url htttps:...../en-US/config and it will show up even if the user is login or not, like so

tungpx · ‎09-18-2024

Hello, In my Splunk web service, we have the domain, for example: https://splunksh.com The problems is anyone can get access to https://splunksh.com/config without login. Although the page doesn't contain any sensitive data, our Cyber Security team deem it as a vulnability that need to be fix. I want to know how to either disable that url, or redirect it to the login page. Any help would be very apriciate.

tungpx · ‎07-19-2024

What if I want to add the requirement that the amount received have to be above 250 and the number of reviced transaction have to be above 10. The original query is index=myindex AMOUNT>=250 |eventstats sum(AMOUNT) as total_sent count as receive by ACCOUNT_FROM |eval temp=ACCOUNT_FROM |where receive >10 |table _time ACCOUNT_TO ACCOUNT_FROM TRACE total_sent INFO temp |join type=inner temp [search index=myindex |stats sum(AMOUNT) as total_received by ACCOUNT_TO |eval temp=ACCOUNT_TO] |where total_sent > total_receive

tungpx · ‎07-19-2024

But it doesn't right, does it? Your query produce total_sent is for ACCOUNT_FROM, and total_received is for ACCOUNT_TO. Since ACCOUNT_FROM and ACCOUNT_TO are two different person then where total_sent > total_received is not make sense.

tungpx · ‎07-19-2024

Hello, I need help with a search query, that at first seem easy but suprising difficult to execute. I have a money transaction db between 2 person, now I have to find which person send out more money than they receive, and output all of their transaction (both send and receive). My query is like so index=myindex |eventstats sum(AMOUNT) as total_sent by ACCOUNT_FROM |eval temp=ACCOUNT_FROM |table _time ACCOUNT_TO ACCOUNT_FROM TRACE total_sent INFO temp |join type=inner temp [search index=myindex |stats sum(AMOUNT) as total_received by ACCOUNT_TO |eval temp=ACCOUNT_TO] |where total_sent > total_receive This query only produce the transaction at which that account sending out but not the transaction that that account receive. How do I go about this. I'm thinking about output the temp as an csv and inputlookup again in the db.

tungpx · ‎07-03-2024

Hello, I have a table in a dashboard like so User ID1 ID2 A ABC 123 B DEF 456 C GHJ 789 I have set a dilldown token like so <set token="id1">$row.ID1$</set> <set token="id1">$row.ID1$</set> I have a below table that I want to pass these token down. For example, when I click at ABC at ID1, it will pass the token id1 aka value "ABC" to below query, when I click at 456 at ID2 it will pass the token id2 aka value "456" to below query. The query will be index=myindex $id1$ (if I click on a value in ID1 column) or index=myindex $id2$ (if I click on a value in ID2 column) I push in the right direction would be very appriciated. Thank you

tungpx · ‎09-06-2023

Hello, I tried to input an DB with query as below: SELECT ..., txn_stamp as TXTIME, .... FROM mybd WHERE txn_stamp > ? ORDER BY TXTIME ASC When I hit Excecute query, the result produce error: ORA-01861: literal does not match format string. My txn_stamp is a time stamp column with the format: YYYY-mm-dd HH:MM:SS (ex: 2023-08-31 00:00:25). The curious thing is sometime it worked, Executing query show data, but it will stop at some point, I suspect it's because of the above error. My thinking is I want to formart either my db timestamp formart or the rising column timestamp formart to the same formart so it won't be a mischatch, but I don't know how.

Posts	12
Solutions	0
Karma Given	2
Karma Received	0
Member Since	‎08-25-2023

Online Status	Offline
Date Last Visited	‎12-27-2024 06:13 AM

Optimizing indexing queue and question about the m...

Log inconsistantly lagging behind

How to check if the input.conf updated on Universa...

How to disable/restrict a certain url

Need help with a search with join, append or appen...

How to pass down token base on row click

Splunk DBconnect 3.14 timestamp as rising column f...