Getting Data In

How to re-ingest data under a different sourcetype?

dionrivera
Path Finder

Recently, I ingested data from a windows event log going back 3 years using the XmlWinEventLog sourcetype. Later, I switched the sourcetype to wineventlog which gave me a easier way to extract fields in the events. I deleted the data and the index and re-created it in hopes of re-ingesting all the events using wineventlog going back 3 years. However. Now, I'm only able to ingest the new events flowing from that event log.  And yes, I am using "ALL TIME" as the timeline.

Is there a way to force Splunk to  scrape everything in an event log?

Labels (1)
Tags (1)
0 Karma
1 Solution

Tom_Lundie
Contributor

Yes, this is possible!

  1. Stop Splunk:
    %SPLUNK_HOME%\bin\splunk.exe stop
  2. Nagivate to the WinEventLog modinputs directory.
    %SPLUNK_HOME%\var\lib\splunk\modinputs\WinEventLog
  3. Delete the files that correspond with the source you want to reset.
  4. Restart Splunk
    %SPLUNK_HOME%\bin\splunk.exe start

Don't forget, you'll only be able to re-index whatever your Windows Host is retaining.

View solution in original post

gcusello
Esteemed Legend

Hi @dionrivera,

in the forwardr used for ingestion, are you ingesting only wineventlogs or also other logs?

if only wineventlogs, you could clean the _fishbucket on the forwarder, but beware because if you do it, you reindex all!

I usually avoid to use this approach!

In my opinion it's easier to maintain the old sourcetype (for the old data not dor the new ones) and use the TA_Windows extractions that works for you, so you haven't to extract any field.

How do you extract fields? are you using TA_Windows or manually?

Ciao.

Giuseppe

0 Karma

dionrivera
Path Finder

@gcusello I am using a separate app to pull the printer logs but I am also collecting windows logs using the same forwarder. I am pulling the logs manually through an app and inputs.conf file. I'll try to delete the fishbucket sub-directory and report back.

0 Karma

Tom_Lundie
Contributor

Yes, this is possible!

  1. Stop Splunk:
    %SPLUNK_HOME%\bin\splunk.exe stop
  2. Nagivate to the WinEventLog modinputs directory.
    %SPLUNK_HOME%\var\lib\splunk\modinputs\WinEventLog
  3. Delete the files that correspond with the source you want to reset.
  4. Restart Splunk
    %SPLUNK_HOME%\bin\splunk.exe start

Don't forget, you'll only be able to re-index whatever your Windows Host is retaining.

dionrivera
Path Finder

@Tom_Lundie Thank you. This resolved my issue. I'm seeing most events using the wineventlog sourcetype. Although, I see some gaps in the data from 2018-2022 and with the XmlWinEventLog sourcetype, I was able to go back to 2017. Any ideas? Thank you again.

Tom_Lundie
Contributor

Glad to hear it, you're welcome!

Is there a gap within a specific source on the same host e.g. XmlWinEventLog:Security has a gap on a specific host?

On the face of it, I can't imagine why a particular eventlog would have a large gap unless the machine has been switched off or that particular eventlog has been disabled.

I would compare what you're seeing in Splunk with what you can see in Windows Event Viewer (particularly for the gap). Could it also be that one the eventlogs has got a large retention set and Splunk is catching up?

0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...