Getting Data In

Why is my indexer randomly indexing old logs?

raynold_peterso
Path Finder

I have noticed that at random times my indexer is indexing old data logs from days, and sometimes even months in the past. I have no clue as to why this is happening. The logs are formatted like this:

1452006410  January 5, 2016 9:06:50 AM CST  NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED    Proview A1827-2100  ATM - A1827-2100 - SERVICEMODE ENTERED  11  DXA CLEAR   Server: INCHARGE-OI
1452006410  January 5, 2016 9:06:50 AM CST  NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED    Proview A1827-2100  ATM - A1827-2100 - SERVICEMODE ENTERED  12  SYSTEM  ESCALATION  MATCHED: Proview2/ArchiveInActiveTraps
1452006410  January 5, 2016 9:06:50 AM CST  NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED    Proview A1827-2100  ATM - A1827-2100 - SERVICEMODE ENTERED  13  SYSTEM  ESCALATION  MATCHED: Notification Clear/Archive - InActive/Archive Inactive Resolved Notifications
1452006410  January 5, 2016 9:06:50 AM CST  NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED    Proview A1827-2100  ATM - A1827-2100 - SERVICEMODE ENTERED  14  SYSTEM  ESCALATION  REACHED: Proview2/ArchiveInActiveTraps, Level-0

At times, I see in the searched logs the date from the indexer will say, this:

1/5/16
9:06:50.000 AM


1448550410 November 26, 2015 9:06:50 AM CST NOTIFICATION-CPU_PerformanceCiscoSystem_I-CPUPerformanceCiscoSystem-PSR-ALBMDSP301/0_HighUtilization CPU_Performance_CiscoSystem I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 HighUtilization 8 SYSTEM ESCALATION SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26, 2015 9:11:51 AM CST
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-Memory
PerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 22 SYSTEM ESCALATION REACHED: Resources/ResoursesClearEvent, Level-1
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-Memory
PerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 23 SYSTEM Action invoked... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-Memory
PerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 24 SYSTEM Action completed successfully... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-Memory
PerformanceHostResources_I-MemoryPerformance_HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 25 SYSTEM Action invoked... zArchiveEvent
Show all 257 lines
ClassName = CPU_Performance_CiscoSystem Escalations = SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26 EventName = HighUtilization InstanceName = I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 SourceEsc = Server: INCHARGE-AM-PM-GA-FL eventtype = ActionSuccess eventtype = Escalations Scheduled eventtype = Notification Clear eventtype = Notification Notify host = ALVIONIX01 source = \ALVIONIX01\d\InCharge\SAM\smarts\local\logs\INCHARGE-SA.audit sourcetype = SAM_Audit

So as you can see, the indexer is picking up older log entries and indexing them as a group as one date.

What can be done?

Any help would be appreciated.

0 Karma

lguinn2
Legend

I don't know that there is "an answer" for this, but I think the following is a pretty good process for figuring it out.

  1. Do all the "odd events" belong to a single host?
  2. What "source" does Splunk show for these events?
  3. If there is more than one source, do all the sources live in the same directory?

And just a thought: do you zip your old log files? Because if you do, that creates a new file. And if that file is in the directory that you are monitoring, Splunk will say "Look! A new file!" and then decompress and index it.

BEST PRACTICE
When you roll old log files, keep the current log file and one prior in the monitored directory.
Zip the older files if you want, but always move them (zipped or not) to a different directory.
Finally the "different directory" should not be a subdirectory of any monitored directory.

Bottom line: I think that old log files may have reappeared in a new location or different format. But they showed up in a directory or subdirectory that Splunk is monitoring...

0 Karma

raynold_peterso
Path Finder

lguinn,

Thanks for the response, but I don't think that is the problem. The system rolls the logs on an almost daily basis. The rolled logs are excluded from indexing so I think I am good there.

The logs its picking up are OLD logs, like months old. Is always only 297 lines every time this happens. The old data is indexed and given todays date with the old 297 lines as a record. The current data is always only one line record from the log file.

So, I still have the issue and don't know how to fix it.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...