topic Re: How to re-ingest data under a different sourcetype in Getting Data In

How to re-ingest data under a different sourcetype?

dionrivera — Thu, 02 Mar 2023 20:20:50 GMT

Recently, I ingested data from a windows event log going back 3 years using the XmlWinEventLog sourcetype. Later, I switched the sourcetype to wineventlog which gave me a easier way to extract fields in the events. I deleted the data and the index and re-created it in hopes of re-ingesting all the events using wineventlog going back 3 years. However. Now, I'm only able to ingest the new events flowing from that event log. And yes, I am using "ALL TIME" as the timeline.

Is there a way to force Splunk to scrape everything in an event log?

Re: How to re-ingest data under a different sourcetype

Tom_Lundie — Tue, 28 Feb 2023 03:28:28 GMT

Yes, this is possible!

Stop Splunk:
%SPLUNK_HOME%\bin\splunk.exe stop
Nagivate to the WinEventLog modinputs directory.
%SPLUNK_HOME%\var\lib\splunk\modinputs\WinEventLog
Delete the files that correspond with the source you want to reset.
Restart Splunk
%SPLUNK_HOME%\bin\splunk.exe start

Don't forget, you'll only be able to re-index whatever your Windows Host is retaining.

Re: How to re-ingest data under a different sourcetype

gcusello — Tue, 28 Feb 2023 08:02:54 GMT

Hi @dionrivera,

in the forwardr used for ingestion, are you ingesting only wineventlogs or also other logs?

if only wineventlogs, you could clean the _fishbucket on the forwarder, but beware because if you do it, you reindex all!

I usually avoid to use this approach!

In my opinion it's easier to maintain the old sourcetype (for the old data not dor the new ones) and use the TA_Windows extractions that works for you, so you haven't to extract any field.

How do you extract fields? are you using TA_Windows or manually?

Ciao.

Giuseppe

Re: How to re-ingest data under a different sourcetype

dionrivera — Thu, 02 Mar 2023 20:10:01 GMT

@gcusello I am using a separate app to pull the printer logs but I am also collecting windows logs using the same forwarder. I am pulling the logs manually through an app and inputs.conf file. I'll try to delete the fishbucket sub-directory and report back.

Re: How to re-ingest data under a different sourcetype

dionrivera — Thu, 02 Mar 2023 22:26:05 GMT

@Tom_Lundie Thank you. This resolved my issue. I'm seeing most events using the wineventlog sourcetype. Although, I see some gaps in the data from 2018-2022 and with the XmlWinEventLog sourcetype, I was able to go back to 2017. Any ideas? Thank you again.

Re: How to re-ingest data under a different sourcetype

Tom_Lundie — Thu, 02 Mar 2023 22:42:20 GMT

Glad to hear it, you're welcome!

Is there a gap within a specific source on the same host e.g. XmlWinEventLog:Security has a gap on a specific host?

On the face of it, I can't imagine why a particular eventlog would have a large gap unless the machine has been switched off or that particular eventlog has been disabled.

I would compare what you're seeing in Splunk with what you can see in Windows Event Viewer (particularly for the gap). Could it also be that one the eventlogs has got a large retention set and Splunk is catching up?