Index skipping similar lines

[OL]

Hello,

Would anyone know how to make sure that splunk index all lines in a file?

The problem I have is that for a file having the following lines

---

I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="14207068", trad_type="SEC", trade_ldt="26-Sep-2011 14:56:25", cp="EXEC", ca="O"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca="" 

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="N", cvi="6000"

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="USD", cvi="6000"

---

is indexed only as:

---

I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca=""

---

I'm using light forwarder (4.2.0) and the inputs.conf file is very simple:

[monitor://<path to log file>.log]
disabled = 0
followTail = 1
sourcetype = DB_CNVW_Log
index = bfm_log

I tried the crcSalt = as well just in case, but didn't help.

On the server side (Splunk 4.2.3), the props.conf is:

[DB_CNVW_Log]
SHOULD_LINEMERGE = false

and there is no transforms.conf for this sourcetype.

Just for information, I'm using 4.2.0 for Light Forwarder because of a bug with wmi.conf in 4.2.1+.

Regards,
Olivier

[Ayn · Accepted solution]

That sounds odd. Are you sure that it's actually not being indexed rather than that it is indexed but that the timestamp is being messed up somehow, so that the messages you're missing might actually be in the index but with incorrect timestamps?

For instance the lines you're missing seem to be referencing the date "26-Sep-2011" and in one case also the date/time "26-Sep-2011 14:56:25". If you perform a search on that time period, do you find the events that are missing there?

[sachinbansal]

@Ayn so how do we resolve this issue?
I am also facing same issue.

[OL]

I think you are spot on!!! Indeed my data was on Sept 26th. Strange that the date recognition isn't the same for all lines! Now I know that this can happen 🙂
Thank you very much.
Olivier

[OL]

Update, it is nothing to do with the Light Forwarder as I tested it on the server itself and I have the same issue.