Splunk not ingesting all logs with xml ... Only ingesting 1 out of every 3

[Strangertinz]

Hi, 

I am dealing with an issue where I am ingesting some logs that contains a few regular line then followed by xml data, but I am only seeing 1 event show up properly with the regular lines and 2 other events get cut short after ingesting the first few lines (examples below). 

So each event is meant to be structured like event1 however they are cut and when I check the actual log file everything is present. 

I tried changing the limits.conf and including maxKBps to 0 but no luck.

[thruput]
maxKBps = 0

Any other ideas as to what could be causing the issue? 



Event1:

2024-11-01 10:04:24,488 23 INFO Sample1 - Customer:11111
ApiKey:xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
DateTime:2024-11-01 10:04:24
RequestBody: <?xml version="1.0" encoding="utf-16"?>........<closing tag>

Event2:

2024-11-01 10:04:26,488 23 INFO Sample1 - Customer:11111
ApiKey:xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Event3: 

2024-11-01 10:04:28,488 23 INFO Sample1 - Customer:11111
ApiKey:xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

[PickleRick]

1. Check your _internal for possible messages regarding this source.

2. Are your sourcetypes properly defined or are you mostly just relying on defaults? I suspect this data source hasn't been properly onboarded. Most importantly - do you have line merging disabled and have properly defined line breaker? (and do you have event breakers set properly?)

3. Did you verify if the rest of those events is really not ingested or maybe just not indexed at the right time? The way to test it would be to run a real-time search (that's one of the very few cases where real-time searches make sense) narrowed down to this problematic source and see whether the data shows up and what timestamp is being parsed from it.

4. Thruput has nothing to do with it. It would only make your downstream pipe get clogged but your data would finally trickle down to the indexer(s).

[isoutamo]

Hi

You could try MC (monitoring console) to look those possible errors in ingestion phase.

Settings -> MC

Indexing -> Inputs -> Data Quality

There are some selections to try to find errors. Then just click those error counts and it will open you query which shows more information about that issue. You could also modify that query to get more information about that issue.

r. Ismo

[sainag_splunk]

Hello @Strangertinz  Have you checked this? 
https://community.splunk.com/t5/Getting-Data-In/Why-is-Windows-event-log-message-data-being-truncate...

Do you have any  other issue with your sourcetype? If this is not working, please work with Splunk support, they might ask you generate a diag with DEBUG options to look out for the TRUNCATE message.

If this Helps, Please UpVote.