topic Re: Index not getting the whole log in Getting Data In

Index not getting the whole log

xisura — Wed, 11 Dec 2013 13:47:29 GMT

Hi Everyone,

I have a problem in indexing of logs. After i search by the source i found out that its not getting the whole content of the log file sample search => index="test" source="sourcepath".
I checked the raw log its complete but in splunk it shows incomplete logs.
Please help how can i fix this?

Thanks,
xisura

Re: Index not getting the whole log

HiroshiSatoh — Wed, 11 Dec 2013 14:21:28 GMT

specification of the log that you want to get the contents of the input.conf correct?

Re: Index not getting the whole log

somesoni2 — Wed, 11 Dec 2013 14:29:25 GMT

Any specific pattern for the missing entries?

Re: Index not getting the whole log

xisura — Wed, 11 Dec 2013 19:43:54 GMT

Hi Here's the sample log

302051908 non-nice user cpu ticks
 67022224 nice user cpu ticks
474810206 system cpu ticks

7723346493 idle cpu ticks
10254021 IO-wait cpu ticks
21190725 IRQ cpu ticks
135816356 softirq cpu ticks
2438955853 interrupts
378813755 CPU context switches
1342633324 boot time
559999898 forks

it cuts here..it didnt index the data below:

DP:
2.13 : 29
3.9 : 29

DPwithFR:
2.2 : 28
2.3 : 30
2.4 : 28
2.5 : 32
2.6 : 30
2.7 : 32

Re: Index not getting the whole log

xisura — Wed, 11 Dec 2013 22:25:51 GMT

hi , the path to specific log in input.conf is correct

Re: Index not getting the whole log

lukejadamec — Wed, 11 Dec 2013 22:38:00 GMT

Is this the expected entire log file, or one event from a log file?
What are you using as a sourcetype?

Re: Index not getting the whole log

xisura — Wed, 11 Dec 2013 22:43:26 GMT

one event per log,i set the sourcetype as "util"

Re: Index not getting the whole log

lukejadamec — Wed, 11 Dec 2013 23:16:52 GMT

What is the sourcetype definition? Can you post the props.conf stanza for [util]?

Re: Index not getting the whole log

lukejadamec — Wed, 11 Dec 2013 23:30:02 GMT

If I was Splunk, this log file would give me a headache. There is no timestamp, the values come before the field sometimes and after the field name other times. However, I am not Splunk.

Regardless, I am new, so what I would do is just index the entire file as raw data and pull out what I wanted from the data with search time field extractions using rex or regex.

To index the entire file without regard to field value content you will need to create a props.conf stanza that never line breaks like this:

props.conf

[util]
SHOULD_LINEMERGE=false
LINE_BREAKER=(?=!)
TRUNCATE=1000000

Reference: Old Folks et.al:

http://answers.splunk.com/answers/11566/how-can-i-index-config-files-and-text-documents-as-individual-events

Re: Index not getting the whole log

xisura — Thu, 12 Dec 2013 00:09:43 GMT

Hi @lukejadamec, should i change the props.conf inside the apps folder or the one inside the systems/local ?

Re: Index not getting the whole log

lukejadamec — Thu, 12 Dec 2013 00:19:59 GMT

I would put it on the indexer in the splunk\etc\system\local\props.conf because it is easier to manage. However, it should also work in the apps \local\props.conf just as well.
If there are any props.conf files anywhere that have a stanza for [util] you need to make sure the settings don't conflict, and I'd restart both the forwarder and indexer splunkd service just to make sure everything is fresh.
Note: This change will only effect logs indexed after the restart.

Re: Index not getting the whole log

lukejadamec — Thu, 12 Dec 2013 00:22:49 GMT

Here is a good read for answering questions regarding where to put/find which config:

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Wheretofindtheconfigurationfiles

Re: Index not getting the whole log

xisura — Thu, 12 Dec 2013 01:00:17 GMT

I already edit the props.conf and restart the indexer and the forwarder still it didnt show that part.

Re: Index not getting the whole log

xisura — Thu, 12 Dec 2013 01:00:36 GMT

I tried searching this sourcetype="util" "DPwithFR:" , set the time to all time, it shows it but the time is "July 19,2013"
I suspect that its not getting the right time so i tried to convert the boot time which is in epoch time to standard time and it shows July 19,2013. Theres no latest data that shows that part , i also checked the raw logs and its there. I dont know why splunk didnt index that part.

1342633324 boot time
565898104 forks
DP:
2.13 : 34
3.9 : 33
DPwithFR:
2.2 : 37
2.3 : 35
2.4 : 36
2.5 : 36
2.6 : 35
2.7 : 35
2.8 : 38

Re: Index not getting the whole log

lukejadamec — Thu, 12 Dec 2013 01:16:36 GMT

Try adding DATETIME_CONFIG = NONE to the props.conf config, and remember this will only affect newly indexed logs after a restart.

For reference:
Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running. When timestamp processing is off, Splunk does not look at the text of the event for the timestamp--it instead uses the event's "time of receipt"; in other words, the time the event is received via its input. For file-based inputs, this means that Splunk derives the event timestamp from the modification time of the input file.

Re: Index not getting the whole log

xisura — Thu, 12 Dec 2013 01:23:51 GMT

ok , i'll keep you updated thanks for helping me 🙂