Splunk Enterprise

Fields aren't being parse correclty

ksbuchanan
Explorer

I am using Universal Forward to collect Windows Security logs from my Domain Controllers. All the logs were being dumped into the "default" (main) index, and we wanted to move to a new index.

I created a new index called "windows". I changed the "c:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" file on the DCs and modified as such:


[default]
host = DCHostName
[WinEventLog://Security]
index=windows

I restarted the Universal Forwarder service. I confirmed that the new events are being written to the new index. That is working correctly.

I wanted to move the "old" logs that has been written to the "main" index to the "windows" index, so I used this command:


index=main AND sourcetype="WinEventLog:Security" | collect index=windows sourcetype="WinEventLog:Security"

I verified that all the logs moved by comparing the count:


(index=main OR index=windows) AND sourcetype="WinEventLog:Security" | stats count(EventCode) by index

Since all the logs, moved, I deleted the logs from the main index"


index=main AND sourcetype="WinEventLog:Security" | delete

However, I discovered several of the fields are being parsed/index/identified correctly. For example, Account_Name is NULL, and Keywords is NULL for all of the logs that were moved from main index to windows index. New logs that are written are being indexed/parsed/identified correctly.

Did I miss a step? shouldn't all of the fields that were moved from the "main" index be indexed in the "windows" index? They were properly index/parsed/identified before I moved from main.

All of my dashboards and reports that were correct previously, are blank or incorrect now - because the field value pairs aren't being properly identified.

Thanks for any help can provide!

0 Karma

ddrillic
Ultra Champion

Just looked at the way we have been doing things with collect and we always use the table command before the collect command, listing the fields we want to move over. Not sure whether table is truly needed...

0 Karma

valiquet
Contributor

|collect write data to indexers without going through the parsing queue. So all index time extractions are gone.

You can configure search time extractions inside props.conf or with |rex

If this do not work, share your props and transforms

0 Karma

hortonew
Builder

I believe by default when using the collect command, your sourcetype becomes "stash" as seen in the documentation for the collect command. The Windows_TA that does search time field extraction by default uses the sourcetype as part of field extraction. Your old data probably has this sourcetype of stash which is why fields aren't being extracted correctly. The new data coming in will have the sourcetype specified at the inputs level.

0 Karma

valiquet
Contributor

No stash when using
| collect index=windows sourcetype="WinEventLog:Security"

0 Karma

hortonew
Builder

Oh woops, missed that part.

0 Karma

ksbuchanan
Explorer

yeah...I also discovered (hard way) that the "host" field was lost during the "move/copy". We only have about 30 days...I'm about to write it off.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.