I have tried to index this file without much success. It's driving me nuts how the fields are never separated correctly no matter what setting I change. I'd be grateful if anyone can try to index this file and let me know what are the props.conf settings you used.
date time field101 field102 field103 field104 field105 field106 field107 field108
2008-10-1 00:15:00.0 35.947 46.170 26.839 26.161 0.079 0.099 0.160 36.833
2008-10-1 00:30:00.0 35.608 46.024 26.791 26.210 0.078 0.098 0.159 36.736
2008-10-1 00:45:00.0 35.608 45.879 26.549 25.871 0.080 0.100 0.162 36.929
2008-10-1 01:00:00.0 35.463 45.976 26.355 25.677 0.081 0.101 0.162 37.057
2008-10-1 01:15:00.0 35.608 45.879 26.452 26.210 0.080 0.100 0.162 36.873
2008-10-1 01:30:00.0 35.754 46.218 26.549 25.967 0.080 0.101 0.163 36.945
2008-10-1 01:45:00.0 35.705 46.170 26.452 26.258 0.081 0.101 0.164 36.949
2008-10-1 02:00:00.0 35.172 45.491 26.549 25.919 0.080 0.101 0.164 37.074
2008-10-1 02:15:00.0 35.415 45.831 26.452 25.919 0.080 0.101 0.163 36.993
2008-10-1 02:30:00.0 35.511 45.637 26.549 25.628 0.082 0.103 0.165 37.170
If these headers will remain statically defined, I would advise against using indexed extractions as it can/will dramatically increase your index size.
I would recommend ingesting each line and then using search time field extractions. That way this awfully formatted file wont have to confirm to CSV and you don't have to worry about using props to massage your data.
Also I don't believe your TIME_FORMAT is correct, if you truly turned it to CSV, the date and time would be comma separated.
I used the following props to ingest with timestamps, then would use a auto field extraction regex to pull the fields.
[ ]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
MAX_DAYS_AGO=10000
I've been at this for hours and totally given up on trying to index it as-is (I think it might be something with the way the number of whitespaces delimiting each value is inconsistent). My latest attempts have been trying to turn it into comma-separated values using SEDCMD, but still no success yet. Below are my settings for this:
[ somedata ]
HEADER_FIELD_LINE_NUMBER=1
CHARSET=UTF-8
MAX_DAYS_AGO=10000
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK=true
SEDCMD-changeintocsv=s/\s+/,/g
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
category=Custom
disabled=false
pulldown_type=true
INDEXED_EXTRACTIONS=csv
The strange thing is I can run regular CLI sed utility on the file to turn it into a CSV file, and Splunk will parse just fine using the csv sourcetype.
I've been at this for hours and totally given up on trying to index it as-is (I think it might be something with the way the number of whitespaces delimiting each value is inconsistent). My latest attempts have been trying to turn it into comma-separated values using SEDCMD, but still no success yet. Below are my settings for this:
[ somedata ]
HEADER_FIELD_LINE_NUMBER=1
CHARSET=UTF-8
MAX_DAYS_AGO=10000
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK=true
SEDCMD-changeintocsv=s/\s+/,/g
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
category=Custom
disabled=false
pulldown_type=true
INDEXED_EXTRACTIONS=csv
The strange thing is I can run regular CLI sed utility on the file to turn it into a CSV file, and Splunk will parse just fine using the csv sourcetype.