I have some json data forward to universal forwarder via syslog. Then universal forwarder is forwarders them to the indexer cluster.
syslog (on log source server) ——> uf——>indexer cluster
When I searching I can see some short json got truncated . It is truncated after about 1000~2000 characters or so, I set props.conf on indexers as follows:
[mysourcetype]
INDEXED_EXTRACTIONS = json
category = Structured
SHOULD_LINEMERGE = false
disabled = false
pulldown_type = true
TIME_FORMAT = %s
TIME_PREFIX = ^\{"timestamp":
TRUNCATE = 0
raw json
{"timestamp":1527213681,"request_headers":{"host":"172.10.101.200:8888","connection":"keep-alive","referer":"http:\/\/172.10.101.200:8888\/superset\/dashboard\/ptsjyy\/?preselect_filters=%7B%22114%22%3A%7B%22__time_grain%22%3A%22month%22%2C%22source%22%3A%5B%5D%7D%7D","accept-encoding":"gzip, deflate, sdch","x-requested-with":"XMLHttpRequest","cookie":"session=.eJyV0N1qxCQABeB3mesQfxONr7KUMNGxhtq6qNtlW_ruFXrdQudu4HxzYD5hj5VaAtfrjSbYzwAOlJFcxKA3iWa13uChtOJSo0WU1iiYwLca915e6G3kjd7sogXxzUfOURgTOQmzbkoRX6TX64FeLmG4XDxmGuYjje2Kz7Sns_VSH-AukHq_OsaEkbNcZsn5LAV3dgwL2NJRsIbXEii_n3RneUA2rvyhOh6Z_iV-7Xma4Nao_nxIwNc3efpdRg.Ddpwlg.tze07woTDVEN_iZ4sIS_jzT4VGI","accept-language":"en-US,en;q=0.8","user-agent":"Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/54.0.2840.71 Safari\/537.36","accept":"application\/json, text\/javascript, *\/*; q=0.01"},"id":"34a2743ad59c93907872","method":"GET","uri":"\/superset\/explore_json\/table\/39\/","client":"10.195.28.22","uri_args":{"form_data":"{\"datasource\":\"39__table\",\"viz_type\":\"area\",\"slice_id\":113,\"granularity_sqla\":\"riqi\",\"time_grain_sqla\":\"month\",\"since\":\"\",\"until\":\"\",\"metrics\":[\"sum__VAL_02\"],\"groupby\":[\"source\"],\"limit\":50,\"timeseries_limit_metric\":null,\"order_desc\":true,\"show_brush\":false,\"show_legend\":true,\"line_interpolation\":\"linear\",\"stacked_style\":\"stack\",\"color_scheme\":\"googleCategory20c\",\"rich_tooltip\":true,\"contribution\":false,\"show_controls\":false,\"x_axis_format\":\"%Y-%m-%d\",\"x_axis_showminmax\":true,\"y_axis_format\":\",\",\"y_axis_bounds\":[null,null],\"y_log_scale\":false,\"rolling_type\":\"None\",\"time_compare\":null,\"num_period_compare\":\"\",\"period_ratio_type\":\"growth\",\"resample_how\":null,\"resample_rule\":null,\"resample_fillmethod\":null,\"annotation_layers\":[],\"where\":\"\",\"having\":\"\",\"filters\":[],\"extra_filters\":[{\"col\":\"__time_grain\",\"op\":\"in\",\"val\":\"month\"},{\"col\":\"source\",\"op\":\"in\",\"val\":[]}]}","preselect_filters":"{\"114\":{\"__time_grain\":\"month\",\"source\":[]}}"},"alerts":[{"msg":"Repetitive non-word characters anomaly detected","id":51002,"match":6}]}
Truncated json (displayed in search results):
{"timestamp":1527213681,"request_headers":{"host":"172.10.101.200:8888","connection":"keep-alive","referer":"http:\/\/172.10.101.200:8888\/superset\/dashboard\/ptsjyy\/?preselect_filters=%7B%22114%22%3A%7B%22__time_grain%22%3A%22month%22%2C%22source%22%3A%5B%5D%7D%7D","accept-encoding":"gzip, deflate, sdch","x-requested-with":"XMLHttpRequest","cookie":"session=.eJyV0N1qxCQABeB3mesQfxONr7KUMNGxhtq6qNtlW_ruFXrdQudu4HxzYD5hj5VaAtfrjSbYzwAOlJFcxKA3iWa13uChtOJSo0WU1iiYwLca915e6G3kjd7sogXxzUfOURgTOQmzbkoRX6TX64FeLmG4XDxmGuYjje2Kz7Sns_VSH-AukHq_OsaEkbNcZsn5LAV3dgwL2NJRsIbXEii_n3RneUA2rvyhOh6Z_iV-7Xma4Nao_nxIwNc3efpdRg.Ddpwlg.tze07woTDVEN_iZ4sIS_jzT4VGI","accept-language":"en-US,en;q=0.8","user-agent":"Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/54.0.2840.71 Safari\/537.36","accept":"application\/json, text\/javascript, *\/*; q=0.01"},"id":"34a2743ad59c93907872","method":"GET","uri":"\/superset\/explore_json\/table\/39\/","client":"10.195.28.22","uri_args":{"form_data":"{\"datasource\":\"39__table\",\"viz_type\":\"area\",\"slice_id\":113,\"granularity_sqla\":\"riqi\",\"time_grain_sqla\":\"month\",\"since\":\"\",\"until\":\"\",\"metrics\":[\"sum__VAL_02\"],\"groupby\":[\"source\"],\"limit\":50,\"timeseries_limit_metric\":null,\"order_desc\":true,\"show_brush\":false,\"show_legend\":true,\"line_interpolation\":\"linear\",\"stacked_style\":\"stack\",\"color_scheme\":\"googleCategory20c\",\"rich_tooltip\":true,\"contribution\":false,\"show_controls\":false,\"x_axis_format\":\"%Y-%m-%d\",\"x_axis_showminmax\":true,\"y_axis_format\":\",\",\"y_axis_bounds\":[null,null],\"y_log_scale\":false,\"rolling_type\":\"None\",\"time_compare\":null,\"num_period_compare\":\"\",\"period_ratio_type\":\"growth\",\"resample_how\":null,\"resample_rule\":null,\"resample_fillmethod\":null,\"annotation_layers\":[],\"where\":\"\",\"having\":\"\",\"filters\":[],\"extra_filters\":[{\"col\":\"__time_grain\",\"op\":\"in\",\"val\":\"month\"},{\"col\":\"source\",\"op\":\"in\",\"val\":[]}]}","preselect_filters":"{\
I use the json validator to verify JSON data syntax and no special characters in JSON.
I try searching internal log . but I can't find any error logs about json truncation.
index=_internal LineBreakingProcessor data_sourcetype=mysourcetype
all help will be appreciated.
... View more