Getting Data In

Json file is getting truncated while uploading

anooshac
Communicator

Hi all,
when i upload a json file to splunk, the data is getting truncated and the full data is not being uploaded. Because of this i'm not getting proper answers for the queries. Can anyone please suggest solutions to avoid this problem.

0 Karma

nickhills
Ultra Champion

Splunk has a default event size of 10,000 bytes. If your Json object is bigger than this it will be truncated before it is parsed.

If your Json block is bigger than this you may need to consider specifying the truncate option in props to configure this to account for your data size.

If my comment helps, please give it a thumbs up!
0 Karma

anooshac
Communicator

[default]
CHARSET = AUTO
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
DATETIME_CONFIG = \etc\datetime.xml
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
HEADER_MODE =
MATCH_LIMIT = 100000
DEPTH_LIMIT = 1000
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
LEARN_MODEL = true
maxDist = 100
AUTO_KV_JSON = true
detect_trailing_nulls = auto
sourcetype =
priority =

Hi @nickhills
I have changed TRUNCATE to 999999 and MAX_EVENTS to 500 but still the data is getting truncated. What will be the problem?

0 Karma

jscraig2006
Communicator

Hi @anooshac,
For large JSON events, i tend to use TRUNCATE=0 to disable truncation. Try that out.
~John

0 Karma

anooshac
Communicator

Hi @jscraig2006, thank you for the response. I changed TRUNCATE to 999999 and restarted the system. The problem got solved now!!

0 Karma

jscraig2006
Communicator

can you post what you have in your props.conf for the sourcetype

0 Karma

anooshac
Communicator

[default]
CHARSET = AUTO
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
DATETIME_CONFIG = \etc\datetime.xml
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
HEADER_MODE =
MATCH_LIMIT = 100000
DEPTH_LIMIT = 1000
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
LEARN_MODEL = true
maxDist = 100
AUTO_KV_JSON = true
detect_trailing_nulls = auto
sourcetype =
priority =

Hello @jscraig2006 thanks for answering. This is the first part of the props.conf file.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...