topic Re: Index skipping similar lines in Getting Data In

Index skipping similar lines

OL — Thu, 29 Sep 2011 14:21:12 GMT

Hello,

Would anyone know how to make sure that splunk index all lines in a file?

The problem I have is that for a file having the following lines

I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="14207068", trad_type="SEC", trade_ldt="26-Sep-2011 14:56:25", cp="EXEC", ca="O"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca="" 

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="N", cvi="6000"

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="USD", cvi="6000"

is indexed only as:

I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca=""

I'm using light forwarder (4.2.0) and the inputs.conf file is very simple:

[monitor://<path to log file>.log]
disabled = 0
followTail = 1
sourcetype = DB_CNVW_Log
index = bfm_log

I tried the crcSalt = as well just in case, but didn't help.

On the server side (Splunk 4.2.3), the props.conf is:

[DB_CNVW_Log]
SHOULD_LINEMERGE = false

and there is no transforms.conf for this sourcetype.

Just for information, I'm using 4.2.0 for Light Forwarder because of a bug with wmi.conf in 4.2.1+.

Regards,
Olivier

Re: Index skipping similar lines

OL — Thu, 29 Sep 2011 15:30:36 GMT

Update, it is nothing to do with the Light Forwarder as I tested it on the server itself and I have the same issue.

Re: Index skipping similar lines

Ayn — Thu, 29 Sep 2011 15:38:49 GMT

That sounds odd. Are you sure that it's actually not being indexed rather than that it is indexed but that the timestamp is being messed up somehow, so that the messages you're missing might actually be in the index but with incorrect timestamps?

For instance the lines you're missing seem to be referencing the date "26-Sep-2011" and in one case also the date/time "26-Sep-2011 14:56:25". If you perform a search on that time period, do you find the events that are missing there?

Re: Index skipping similar lines

OL — Thu, 29 Sep 2011 16:02:45 GMT

I think you are spot on!!! Indeed my data was on Sept 26th. Strange that the date recognition isn't the same for all lines! Now I know that this can happen 🙂
Thank you very much.
Olivier

Re: Index skipping similar lines

sachinbansal — Tue, 23 Jul 2019 10:08:13 GMT

@Ayn so how do we resolve this issue?
I am also facing same issue.