Fields aren't being parse correclty

[ksbuchanan]

I am using Universal Forward to collect Windows Security logs from my Domain Controllers. All the logs were being dumped into the "default" (main) index, and we wanted to move to a new index.

I created a new index called "windows". I changed the "c:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" file on the DCs and modified as such:

    [default]

    host = DCHostName

    [WinEventLog://Security]

    index=windows

I restarted the Universal Forwarder service. I confirmed that the new events are being written to the new index. That is working correctly.

I wanted to move the "old" logs that has been written to the "main" index to the "windows" index, so I used this command:

index=main  AND sourcetype="WinEventLog:Security" | collect index=windows sourcetype="WinEventLog:Security"

I verified that all the logs moved by comparing the count:

(index=main  OR index=windows) AND sourcetype="WinEventLog:Security" | stats count(EventCode) by index

Since all the logs, moved, I deleted the logs from the main index"

index=main  AND sourcetype="WinEventLog:Security" | delete

However, I discovered several of the fields are being parsed/index/identified correctly. For example, Account_Name is NULL, and Keywords is NULL for all of the logs that were moved from main index to windows index. New logs that are written are being indexed/parsed/identified correctly.

Did I miss a step? shouldn't all of the fields that were moved from the "main" index be indexed in the "windows" index? They were properly index/parsed/identified before I moved from main.

All of my dashboards and reports that were correct previously, are blank or incorrect now - because the field value pairs aren't being properly identified.

Thanks for any help can provide!

[ddrillic]

Just looked at the way we have been doing things with collect and we always use the table command before the collect command, listing the fields we want to move over. Not sure whether table is truly needed...

[valiquet]

|collect write data to indexers without going through the parsing queue. So all index time extractions are gone.

You can configure search time extractions inside props.conf or with |rex

If this do not work, share your props and transforms

[hortonew]

I believe by default when using the collect command, your sourcetype becomes "stash" as seen in the documentation for the collect command. The Windows_TA that does search time field extraction by default uses the sourcetype as part of field extraction. Your old data probably has this sourcetype of stash which is why fields aren't being extracted correctly. The new data coming in will have the sourcetype specified at the inputs level.

[valiquet]

No stash when using
| collect index=windows sourcetype="WinEventLog:Security"

[hortonew]

Oh woops, missed that part.

[ksbuchanan]

yeah...I also discovered (hard way) that the "host" field was lost during the "move/copy". We only have about 30 days...I'm about to write it off.