How to resolve timestamp and line processing issues in pdfgen.log ?

[damode]

I am getting the below two warning messages,
1. 11-27-2017 06:00:22.902 +1100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Mon Nov 27 06:00:20 2017). Context: source::C:\Program Files\Splunk\var\log\splunk\pdfgen.log|host::INDEXER|splunk_pdfgen|20662

11-27-2017 06:00:16.835 +1100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 17586 - data_source="C:\Program Files\Splunk\var\log\splunk\pdfgen.log", data_host="INDEXER", data_sourcetype="splunk_pdfgen"

1. Sample timestamp in pdfgen.log looks like this
   2017-11-27 06:01:00,206 +1100 INFO pdfgen_table:1041 - renderTable> headerRow: ['host', 'src_interface', 'port_status', 'count']
   2017-11-27 06:01:09,519 +1100 INFO pdfgen_endpoint:271 - Generated pdf, filename = overview-2017-11-27.pdf

2. props.conf
   [splunk_pdfgen]
   TIME_FORMAT = %m-%d-%Y %H:%M%S,%l
   SHOULD_LINEMERGE = False
   MAX_TIMESTAMP_LOOKAHEAD = 40

[arekdabrowski]

I have the same problem on version 7.3.1
When I have the default props.conf file in the pdfgen file, my data quality displays problems with timestamp analysis, here are the details:
01-15-2020 11:56:18.641 +0100 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Wed Jan 15 11:56:15 2020). Context: source=/opt/splunk/var/log/splunk/pdfgen.log|host=xxxxxxxxxxxxx|splunk_pdfgen|2557
When I add to my props.conf on the system / local / props.conf TIME_FORMAT =% Y-% m-% d% H:% M:% S,% 3N% z
I also have the same problem.
Do you have any ideas?

[MuS]

Hi damode,

the TIME_FORMAT = %m-%d-%Y %H:%M%S,%l should be TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N.
Regarding the truncating add TRUNCATE = 20000 to the props.conf

Hope this helps ...

cheers, MuS

[damode]

Hi @MuS,

Thanks for your prompt reply.

I have applied the suggested settings. Will let you know the outcome.

Regards,
Dev

[damode]

Hi @MuS,

I am not getting Truncating line issue anymore. Thanks for that! I am still, however, getting the timestamp issues.

1. 11-28-2017 06:00:16.854 +1100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Tue Nov 28 06:00:14 2017). Context: source::C:\Program Files\Splunk\var\log\splunk\pdfgen.log|host::INDEXER|splunk_pdfgen|126
2. props.conf [splunk_pdfgen] TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N SHOULD_LINEMERGE = False MAX_TIMESTAMP_LOOKAHEAD = 40 TRUNCATE = 20000

[MuS]

I just checked the default settings for [splunk_pdfgen] and it actually has this option set:

 TIME_FORMAT = %m-%d-%Y %H:%M%S,%l

So, please remove the TIME_FORMAT you added and try again - really wired...

Can you run this command /opt/splunk/bin/splunk btool props list splunk_pdfgen --debug and compare to this list of options please:

/opt/splunk/etc/system/default/props.conf                  [splunk_pdfgen]
/opt/splunk/etc/system/default/props.conf                  ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf                  ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                  AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                  BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                  BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                  CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                  DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                  HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                  LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                  LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                  LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                  MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                  MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                  MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                  MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                  MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                  MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                  MAX_TIMESTAMP_LOOKAHEAD = 40
/opt/splunk/etc/system/default/props.conf                  MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                  MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                  MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                  SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                  SHOULD_LINEMERGE = False
/opt/splunk/etc/system/default/props.conf                  TIME_FORMAT = %m-%d-%Y %H:%M%S,%l
/opt/splunk/etc/system/default/props.conf                  TRANSFORMS = 
/opt/splunk/etc/system/default/props.conf                  TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                  detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                  maxDist = 100
/opt/splunk/etc/system/default/props.conf                  priority = 
/opt/splunk/etc/system/default/props.conf                  sourcetype = 

[damode]

Hi @MuS,

Upon comparing with the above list of options, I found the below fields having different value in comparison to yours. Everything else is same.
CHARSET = AUTO
TRUNCATE = 20000
detect_trailing_nulls = auto

[damode]

Hi @MuS, I had changed back to default TIME_FORMAT, but that still gave the same issue.
Based on the above observation, do you recommend setting the [splunk_pdfgen] attributes exactly same as yours ?

[MuS]

Well, the above settings are the Splunk default settings so they really should work.

[damode]

Now I am getting the same error from datasourcetype = licensealert-5 as well, in addition to splunk_pdfgen.

[MuS]

That sounds like a bigger problem here .... also reading all you other questions.

Random question: have you done a FS check lately on your Splunk server to see if everything is healthy?

[damode]

If you mean health check on DMC, then yes.
On Search head, I have license warning and scheduled searches skipped messages. On Indexer, I am getting these event processing issue about which I have posted here.

[MuS]

No I meant an actual file system check from the operating system.

[damode]

Hi @MuS, for some reason, the Search Head had the same hostname as the Indexer. Not sure how and when I did that. Once I changed it to its correct username, I stopped getting time parsing warning messages. I believe, that’s probably what was causing the issue.

[damode]

I just did a file system check from the operating system using SFC.EXE /scannow and did not find any integrity violations.