Getting Data In

How to re-ingest data under a different sourcetype?

dionrivera
Path Finder

Recently, I ingested data from a windows event log going back 3 years using the XmlWinEventLog sourcetype. Later, I switched the sourcetype to wineventlog which gave me a easier way to extract fields in the events. I deleted the data and the index and re-created it in hopes of re-ingesting all the events using wineventlog going back 3 years. However. Now, I'm only able to ingest the new events flowing from that event log.  And yes, I am using "ALL TIME" as the timeline.

Is there a way to force Splunk to  scrape everything in an event log?

Labels (1)
Tags (1)
0 Karma
1 Solution

Tom_Lundie
Contributor

Yes, this is possible!

  1. Stop Splunk:
    %SPLUNK_HOME%\bin\splunk.exe stop
  2. Nagivate to the WinEventLog modinputs directory.
    %SPLUNK_HOME%\var\lib\splunk\modinputs\WinEventLog
  3. Delete the files that correspond with the source you want to reset.
  4. Restart Splunk
    %SPLUNK_HOME%\bin\splunk.exe start

Don't forget, you'll only be able to re-index whatever your Windows Host is retaining.

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @dionrivera,

in the forwardr used for ingestion, are you ingesting only wineventlogs or also other logs?

if only wineventlogs, you could clean the _fishbucket on the forwarder, but beware because if you do it, you reindex all!

I usually avoid to use this approach!

In my opinion it's easier to maintain the old sourcetype (for the old data not dor the new ones) and use the TA_Windows extractions that works for you, so you haven't to extract any field.

How do you extract fields? are you using TA_Windows or manually?

Ciao.

Giuseppe

0 Karma

dionrivera
Path Finder

@gcusello I am using a separate app to pull the printer logs but I am also collecting windows logs using the same forwarder. I am pulling the logs manually through an app and inputs.conf file. I'll try to delete the fishbucket sub-directory and report back.

0 Karma

Tom_Lundie
Contributor

Yes, this is possible!

  1. Stop Splunk:
    %SPLUNK_HOME%\bin\splunk.exe stop
  2. Nagivate to the WinEventLog modinputs directory.
    %SPLUNK_HOME%\var\lib\splunk\modinputs\WinEventLog
  3. Delete the files that correspond with the source you want to reset.
  4. Restart Splunk
    %SPLUNK_HOME%\bin\splunk.exe start

Don't forget, you'll only be able to re-index whatever your Windows Host is retaining.

dionrivera
Path Finder

@Tom_Lundie Thank you. This resolved my issue. I'm seeing most events using the wineventlog sourcetype. Although, I see some gaps in the data from 2018-2022 and with the XmlWinEventLog sourcetype, I was able to go back to 2017. Any ideas? Thank you again.

Tom_Lundie
Contributor

Glad to hear it, you're welcome!

Is there a gap within a specific source on the same host e.g. XmlWinEventLog:Security has a gap on a specific host?

On the face of it, I can't imagine why a particular eventlog would have a large gap unless the machine has been switched off or that particular eventlog has been disabled.

I would compare what you're seeing in Splunk with what you can see in Windows Event Viewer (particularly for the gap). Could it also be that one the eventlogs has got a large retention set and Splunk is catching up?

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Hi Splunky people! We are excited to share the newest updates in Splunk Enterprise 9.3!Admins and Analyst can ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...