Getting Data In

Monitor Log that changes the first few characters every few minutes causing duplicate indexing of the same log.

gorgiea
Loves-to-Learn

Hello, 

Trying to monitor a log which changes the first few characters of the log every few minutes, this seems to cause Splunk UF to re-index the whole log each time, since it see's the first few characters as being different which causes a difference crc. 

I have tried many different options but nothing seems to be working properly to index this log properly and without duplicates. Example of log below.

 

.log  at 02:53

eÎ5  eÎ5   014500000000000003FGR0002TRA00102021/01/24001202:53 32.0850006

same .log a few minutes later at 02:56

ØT&  ØT&   014500000000000003FGR0002TRA00102021/01/24001202:53 32.0850006
014500000000000003FGR0002TRA00102021/01/24001202:55 42.0150006
014500000000000003FGR0002TRA00102021/01/24001202:56 33.0110006

 

Seems the App that generates this log changes these characters for some reason and there is no way to capture the data without getting duplicate data. Any suggestions? Thanks

 

 

Labels (3)
Tags (2)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Supercharging Windows Security Detection Performance: Introducing Hybrid Field ...

Windows event logs—from Security auditing and Sysmon to PowerShell script blocks—form the operational backbone ...

Ditch the Manual Grind: Building AI Agents with Splunk

Ditch the Manual Grind: Building AI Agents with Splunk Let’s be real: your team’s time is being eaten alive. ...

Cisco Data Fabric from Architecture to Investigation, Better SOC Visibility, and More ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...