Hi @PickleRick, Do you any clue for fix with any other possible way workaround ? ... View more

@PickleRick  As per the link you shared at first place. There mentioned that remove 'grandableRoles' =admin  from the admin user from authorize.conf file. Is that workaround will work or shall I try ? ... View more

Hi @PickleRick , I am facing the similar issue.  Exactly I need to do ?  shall I change 'grandableRoles' =admin in autoritize.conf file ? ... View more

@PickleRick I mean to say. The value of the TASKIDUPDATED field is always unique value after applying checkpoint value event should be ingested only once and not multiple times.   Below is the setting I am currently using for db connect.  connection = VIn disabled = 0 index = group_data index_time_mode = current interval = */10 * * * * max_rows = 0 mode = rising query = SELECT * FROM "WMCDB"."KLDGSF_ROUPOVERVIEW"\ WHERE TASKIDUPDATED < ?\ ORDER BY TASKIDUPDATED DESC query_timeout = 30 sourcetype = overview_packgroup tail_rising_column_init_ckpt_value = {"value":null,"columnType":null} tail_rising_column_name = TASKIDUPDATED tail_rising_column_number = 3 input_timestamp_column_number = 10 input_timestamp_format = ... View more

@PickleRick I am using field name "TASKIDUPDATED" which is the combination of TASKID and UPDATED column and it is always dynamic in nature. I have given this field in the rising column and this field is changing in every run. Even after this, duplicate data is being ingested. ... View more

My database contains two types of events, and I want to ensure that only the latest row for each unique TASKID is ingested into Splunk with the following requirements: Latest Status: Only the most recent status for each TASKID should be captured, determined by the UPDATED timestamp field. Latest Date: The row with the most recent UPDATED timestamp for each TASKID should be ingested into Splunk. Single Count: Each TASKID should appear only once in Splunk, with no duplicates or older rows included. Please help me achieve this requirement. Currently method I am using is "Rising column update" method. But still splunk is not ingesting a row with the latest status. I am using below query in SQL input under DB connect. SELECT * FROM "DB"."KSF_OVERVIEW" WHERE TASKIDUPDATED > ? ORDER BY TASKIDUPDATED ASC   Below are the sample events from the database. =====Status "FINISHED" 2024-12-06 11:50:22.984, TASKID="11933815411", TASKLABEL="11933815411", TASKIDUPDATED="11933815411 2024/12/05 19:40:47", TASKTYPEKEY="PACKGROUP", CREATED="2024-12-05 14:18:18", UPDATED="2024-12-05 19:40:47", STATUSTEXTKEY="Dynamic|TaskStatus.key{FINISHED}.textKey", CONTROLLERSTATUSTEXTKEY="Dynamic|TaskControllerStatus.taskTypeKey{PACKGROUP},key{EXECUTED}.textKey", STATUS="FINISHED", CONTROLLERSTATUS="EXECUTED", REQUIREDFINISHTIME="2024-12-06 00:00:00", STATION="PAL/Pal02", REQUIRESCUBING="0", REQUIRESQUALITYCONTROL="0", PICKINGSUBTASKCOUNT="40", TASKTYPETEXTKEY="Dynamic|TaskType.Key{PACKGROUP}.textKey", OPERATOR="1", MARSHALLINGTIME="2024-12-06 06:30:00", TSU="340447278164799274", FMBARCODE="WMC000000000341785", TSUTYPE="KKP", TOURNUMBER="2820007682", TYPE="DELIVERY", DELIVERYNUMBER="17620759", DELIVERYORDERNUMBER="3372948211", SVSSTATUS="DE_FINISHED", STORENUMBER="0000002590", STACK="11933816382", POSITION="Bottom", LCTRAINID="11935892717", MARSHALLINGAREA="WAB" =====Status "RELEASED" 2024-12-05 14:20:13.290, TASKID="11933815411", TASKLABEL="11933815411", TASKIDUPDATED="11933815411 2024/12/05 14:18:20", TASKTYPEKEY="PACKGROUP", CREATED="2024-12-05 14:18:18", UPDATED="2024-12-05 14:18:20", STATUSTEXTKEY="Dynamic|TaskStatus.key{RELEASED}.textKey", CONTROLLERSTATUSTEXTKEY="Dynamic|TaskControllerStatus.taskTypeKey{PACKGROUP},key{CREATED}.textKey", STATUS="RELEASED", CONTROLLERSTATUS="CREATED", REQUIREDFINISHTIME="2024-12-06 00:00:00", REQUIRESCUBING="0", REQUIRESQUALITYCONTROL="0", PICKINGSUBTASKCOUNT="40", TASKTYPETEXTKEY="Dynamic|TaskType.Key{PACKGROUP}.textKey", OPERATOR="1", MARSHALLINGTIME="2024-12-06 06:30:00", TSUTYPE="KKP", TOURNUMBER="2820007682", TYPE="DELIVERY", DELIVERYNUMBER="17620759", DELIVERYORDERNUMBER="3372948211", SVSSTATUS="DE_CREATED", STORENUMBER="0000002590", STACK="11933816382", POSITION="Bottom", MARSHALLINGAREA="WAB" ... View more

Hi @bowesmana @PickleRick just for both of your information. When I replaced endpoint /services/collector/event?auto_extract_timestamp=true with /services/collector/raw?auto_extract_timestamp=true, correct raw data format started coming and the timestamp also started matching .  Example as below. Thanks both of your support and valuable suggestions.   ... View more

Hi @bowesmana, Events are not showing as expected after selecting "show source".   ... View more

Hi @PickleRick, If I replace the TASKID column with UPDATED column to rising column method, will it make a difference?  FYI : I also increased the checkpoint value from 1 to 2 and even after the second time STATUS change is RELEASED to FINISHED, that row is not ingested in splunk. ... View more

Is someone can support me on this topic ? ... View more

Hi @PickleRick, Just to inform you. I am using below props setting in my Prod env. But still I can see discrepancies.  DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = 1 TIME_PREFIX = \"@timestamp\" TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z   ... View more

Hi @PickleRick  I have corrected transforms.conf from [add_time] INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ") to   [add_time] INGEST_EVAL = _time=strftime(strptime(_time, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")   Note : In my opinion, the parsing of the timestamp should be correct first so that we may convert using INGEST_EVAL. In my case, the time format ("%Y-%m-%dT%H:%M:%S.%9N%:z") is not parsing properly, which may cause an issue during timestamp conversion. ... View more

Hi @PickleRick, I have tried below workaround but still timestamp is not converting as per my requirement. props.conf [timestamp_change] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = 1 TIME_PREFIX = \"@timestamp\" TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%:z TRANSFORMS-add_time = add_time transforms.conf [add_time] INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ") ... View more

Thanks @bowesmana for showcasing this example. So its that okay to skip micro seconds and good to use %9N in real time data flow? ... View more

@meetmshah  Thanks for your suggestion. I will try it definitely   Meanwhile before your suggested workaround. I have tried myself with INGEST_EVAL attribute in transforms.conf with props.conf and fields.conf and it is working.   ... View more

Hi @PickleRick In this case, can I skip to use TIME_FORMAT? Is TIME_PREFIX and linebreaker attributes is enough here ? ... View more

Hi @PickleRick  just to inform you. I have replaced below endpoint but still the mismatch of the timestamp issue persist.    ... View more

Hi @PickleRick  I am using this endpoint "/services/collector" And how I can explicitly use timestamp format in the endpoint ?   ... View more

@gcusello As per the below screenshot, I need to specify in the match_type for both the fields ? FYI @gcusello  I have added below entries and it starts working as expected. WILDCARD(source), WILDCARD(position), WILDCARD(destination) ... View more

@bowesmana Thanks for the solution and investing your valuable time.  But still micro seconds are not matching.        ... View more