Stack trace data truncated when indexed. Search shows some data or punctuations missing.

[Ellen]

We are running Splunk 4.3.4.

Here is a sample stack trace from the server along with the results from a search for the same source.

To sum up what we are seeing, at line 70, Splunk starts removing periods and underscores.
It also removed common parts of the stack trace.

This is making the stack trace very hard to read and use for troubleshooting issues.

Below are sample lines from the actual stack trace:

at org.apache.yoyo.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[yoyo.jar:6.0.29]
at com.abc.common.filter.VisitPageEventPublishFilter.doFilterInternal(VisitPageEventPublishFilter.java:44) ~[xyz_web_app.jar:13.11]
at com.zzz.xyz.common.web.filter.SpringManagedFilter.doFilter(SpringManagedFilter.java:97) ~[hcom-websupport-11.5.jar:11.5]

Below are the search results of the same data set for the stack trace:

at org.apache.yoyo.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[:.yoyojar6.0.29]
at ...() [..comabccommonfilterVisitPageEventPublishFilterdoFilterInternal:.VisitPageEventPublishFilterjava44~.:.]
__xyzwebappjar1311at.....() [..comzzzxyzcommonwebfilterSpringManagedFilterdoFilter:.SpringManagedFilterjava97~..:.]

You can see, it starts removing the periods and underscores along with common parts of the line.

We have indexed the attached stack trace into a local instance of Splunk and it is not showing the same problems when we run a search against it.

We have also confirmed that the only entry we have in the prop.conf or transform.conf files is for the max events setting which is copied below. We setup the same setting on our local instance and re-index the stack trace without any issues, so we do no think it is the cause of the issues.

[hwa_app]
MAX_EVENTS=1024 

How can we troubleshoot?

[Ellen · Accepted solution]

First verify what you see in the browser is really how Splunk indexed the data.

A quick check to ensure this was not a browser issue is to run the same search under another browser.

Verify the actual data events indexed are the same as the raw by either
1) selecting 'Export' in any browser or
2) run the search via CLI to view the results.

This was a browser issue as it was confirmed the data was indexed properly.

Under Safari Version 6.0.2 (7536.26.17) the issue was reproducible and the results appear to be mangled.
There were some earlier versions of Chrome where this was also reproduced.

This is a known bug related to Webkit and no longer an issue in Splunk 5.0 since the event viewer has changed. (SPL-55380/SPL-55354)

So check what browsers and version are in use and apply the workarounds below.

The following browser versions could not reproduce the issue and displayed as expected.

IE 9.0.8112.16421
FF 17.0.1
Chrome Version 23.0.1271.95

Workarounds:
1) try using another browser or a more recent version of Firefox, Chrome, or IE.
or
2) you can try to work in the search results UI, select the 'Option' > this will display a 'Results Display Options' window and one of the display options is to select the event segmentation. By default it is 'full' but if you change it to outer or inner, you will not see the odd results.
Don't confuse this with index time segmentation. This workaround is for the search time display.

http://docs.splunk.com/Documentation/Splunk/4.3.4/Data/SetsegmentationinSplunkWeb