Getting Data In

Index skipping similar lines

OL
Communicator

Hello,

Would anyone know how to make sure that splunk index all lines in a file?

The problem I have is that for a file having the following lines


I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="14207068", trad_type="SEC", trade_ldt="26-Sep-2011 14:56:25", cp="EXEC", ca="O"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca="" 

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="N", cvi="6000"

I:11-09-29 14:58:23:Posn ld="26-SEP-2011", tt="SEC", bk="41208", str="NONE", cont="-126", subcont="-126", isec="8274", ccy="GBP", cp="EXEC", pt="USD", cvi="6000"

is indexed only as:


I:11-09-29 14:58:23:Processing P&L on "T14207068"

I:11-09-29 14:58:23:Getting CVL AK for trade cont_id="", trad_type="", trade_ldt="", cp="", ca=""

I'm using light forwarder (4.2.0) and the inputs.conf file is very simple:

[monitor://<path to log file>.log]
disabled = 0
followTail = 1
sourcetype = DB_CNVW_Log
index = bfm_log

I tried the crcSalt = as well just in case, but didn't help.

On the server side (Splunk 4.2.3), the props.conf is:

[DB_CNVW_Log]
SHOULD_LINEMERGE = false

and there is no transforms.conf for this sourcetype.

Just for information, I'm using 4.2.0 for Light Forwarder because of a bug with wmi.conf in 4.2.1+.

Regards,
Olivier

Tags (2)
0 Karma
1 Solution

Ayn
Legend

That sounds odd. Are you sure that it's actually not being indexed rather than that it is indexed but that the timestamp is being messed up somehow, so that the messages you're missing might actually be in the index but with incorrect timestamps?

For instance the lines you're missing seem to be referencing the date "26-Sep-2011" and in one case also the date/time "26-Sep-2011 14:56:25". If you perform a search on that time period, do you find the events that are missing there?

View solution in original post

Ayn
Legend

That sounds odd. Are you sure that it's actually not being indexed rather than that it is indexed but that the timestamp is being messed up somehow, so that the messages you're missing might actually be in the index but with incorrect timestamps?

For instance the lines you're missing seem to be referencing the date "26-Sep-2011" and in one case also the date/time "26-Sep-2011 14:56:25". If you perform a search on that time period, do you find the events that are missing there?

sachinbansal
New Member

@Ayn so how do we resolve this issue?
I am also facing same issue.

0 Karma

OL
Communicator

I think you are spot on!!! Indeed my data was on Sept 26th. Strange that the date recognition isn't the same for all lines! Now I know that this can happen 🙂
Thank you very much.
Olivier

0 Karma

OL
Communicator

Update, it is nothing to do with the Light Forwarder as I tested it on the server itself and I have the same issue.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...