I have a problem in indexing of logs. After i search by the source i found out that its not getting the whole content of the log file sample search => index="test" source="sourcepath".
I checked the raw log its complete but in splunk it shows incomplete logs.
Please help how can i fix this?
If I was Splunk, this log file would give me a headache. There is no timestamp, the values come before the field sometimes and after the field name other times. However, I am not Splunk.
Regardless, I am new, so what I would do is just index the entire file as raw data and pull out what I wanted from the data with search time field extractions using rex or regex.
To index the entire file without regard to field value content you will need to create a props.conf stanza that never line breaks like this:
[util] SHOULD_LINEMERGE=false LINE_BREAKER=(?=!) TRUNCATE=1000000
Reference: Old Folks et.al:
DATETIME_CONFIG = NONE to the props.conf config, and remember this will only affect newly indexed logs after a restart.
Set DATETIME_CONFIG = NONE to prevent the timestamp processor from running. When timestamp processing is off, Splunk does not look at the text of the event for the timestamp--it instead uses the event's "time of receipt"; in other words, the time the event is received via its input. For file-based inputs, this means that Splunk derives the event timestamp from the modification time of the input file.
I tried searching this sourcetype="util" "DPwithFR:" , set the time to all time, it shows it but the time is "July 19,2013"
I suspect that its not getting the right time so i tried to convert the boot time which is in epoch time to standard time and it shows July 19,2013. Theres no latest data that shows that part , i also checked the raw logs and its there. I dont know why splunk didnt index that part.
1342633324 boot time
2.13 : 34
3.9 : 33
2.2 : 37
2.3 : 35
2.4 : 36
2.5 : 36
2.6 : 35
2.7 : 35
2.8 : 38
I would put it on the indexer in the splunk\etc\system\local\props.conf because it is easier to manage. However, it should also work in the apps \local\props.conf just as well.
If there are any props.conf files anywhere that have a stanza for
[util] you need to make sure the settings don't conflict, and I'd restart both the forwarder and indexer splunkd service just to make sure everything is fresh.
Note: This change will only effect logs indexed after the restart.
Hi Here's the sample log
302051908 non-nice user cpu ticks 67022224 nice user cpu ticks 474810206 system cpu ticks
7723346493 idle cpu ticks
10254021 IO-wait cpu ticks
21190725 IRQ cpu ticks
135816356 softirq cpu ticks
378813755 CPU context switches
1342633324 boot time
it cuts here..it didnt index the data below:
2.13 : 29
3.9 : 29
2.2 : 28
2.3 : 30
2.4 : 28
2.5 : 32
2.6 : 30
2.7 : 32