Upgrade and My Dashboard is broke from DataInputs/Fields

[jenkinsta]

I upgraded a minor version recently and my data inputs and field extractions are removed. So my dashboard no longer works. Is this normal for upgrades? Also how can I link them back so in the dashboards or the search the fields are properly extracted?

[richgalloway]

That's not normal unless you created your data inputs and field extractions by modifying files in $SPLUNK_HOME/etc/system/default.
What did you upgrade from and to?

[jenkinsta]

I migrated to VERSION=8.0.5 from the previous version I downloaded in May 2020. 

Nothing was unique from this installation. I had the trial and now the free limited version which I upgraded the same time my trial expired and got my free version license. 

I have several folders on my linux box with data inputs for some logging like my current wifi, current temp, remote ip, internet speed. I used the input wizard and extracted fields then took that search and created dashboard panels. After the upgrade and licence change it got disconnected.  

[isoutamo]

This sound weird.

Can you see if there is something related to this on migration.log-xxxx.xxx file under var/log/splunk.

r. Ismo

[jenkinsta]

migration.log.2020-07-30.12-07-30

[jenkinsta]

Migrating to:
VERSION=8.0.5
BUILD=a1a6394cc5ae
PRODUCT=splunk
PLATFORM=Linux-x86_64

Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Checking for possible timezone configuration errors...

Handling deprecated files...

Checking script configuration...

Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default' to '/opt/splunk/etc/myinstall/splunkd.xml'.
Deleting '/opt/splunk/etc/system/local/field_actions.conf'.

The following apps might contain lookup table files that are not exported to other apps:

departures-board-viz
event-timeline-viz
heat-map-viz
missile_map
network-diagram-viz
splunk_monitoring_console

Such lookup table files could only be used within their source app. To export them globally and allow other apps to access them, add the following stanza to each /opt/splunk/etc/apps/<app_name>/metadata/local.meta file:

[lookups]
export = system

For more information, see http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/SetPermissions#Make_objects_globally_....

Checking for possible UI view conflicts...
App "splunk_monitoring_console" has an overriding copy of the "dashboards.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views
App "splunk_monitoring_console" has an overriding copy of the "reports.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views
App "splunk_monitoring_console" has an overriding copy of the "alerts.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views
Removing legacy manager XML files...
Removing legacy nav XML files...
DMC is not set up, no need to migrate nav bar.
Removing System Activity dashboards...
Removing splunkclouduf XML file...
Removing splunkclouduf view XML files...
Distributed Search is not configured on this instance
Removing legacy search.xml file from splunk_instrumentation...
Deleting '/opt/splunk/share/splunk/search_mrsparkle/modules'.
Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Checking for the modules related files and folders that should not be present after upgrade.

Checking for the Advanced XML dashboard templates that should not be present after upgrade.

Checking for the 'Getting Started' app that should not be present after upgrade.

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.

"/opt/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal
"/opt/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal
Clustering migration already complete, no further changes required.

Generating checksums for datamodel and report acceleration bucket summaries for all indexes.
If you have defined many indexes and summaries, summary checksum generation may take a long time.
Processed 1 out of 12 configured indexes.
Processed 2 out of 12 configured indexes.
Processed 3 out of 12 configured indexes.
Processed 4 out of 12 configured indexes.
Processed 5 out of 12 configured indexes.
Processed 6 out of 12 configured indexes.
Processed 7 out of 12 configured indexes.
Processed 8 out of 12 configured indexes.
Processed 9 out of 12 configured indexes.
Processed 10 out of 12 configured indexes.
Processed 11 out of 12 configured indexes.
Processed 12 out of 12 configured indexes.
Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
[App Key Value Store migration] Migration is not required.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600 seconds.
[App Key Value Store migration] Migration is not required.
[DFS] Performing migration.
[DFS] Finished migration.

[isoutamo]

Unfortunately at least I cannot see anything special which related to your problem.

One another place which you could check is root's history if there is any commands wihch can explain this? Otherwise it's hard (/impossible) said what was caused that unnormal behaviour.

r. Ismo

[jenkinsta]

Ok, thanks for the help. Not a big deal if a one off. But I don't want to rebuild every time. 

[richgalloway]

Don't rebuild - restore your latest backup.
If you put your dashboards and other knowledge objects in a custom app then backup and restore of that app is trivial. It's also a recommended part of the upgrade process.

[jenkinsta]

I think I see the problem. During the trial I created these things under a user i created or admin. Since I downgraded there is no user. I changed all the permissions/owner to nobody but not sure where else I need to change. But my newly created items are listed as nobody as the owner. 

[richgalloway]

That should be easy to fix. Go to the CLI and move files from $SPLUNK_HOME/etc/users/<user> to $SPLUNK_HOME/etc/users/admin. Then restart Splunk.

[isoutamo]

That’s true. In trial you could use several users and another features which is not usable on free version. That’s said on documentation too. 
In your case you probably found those KOs on disk if you look those on under etc.

r. Ismo

[isoutamo]

No this is not a normal. Basically there are two reason what came my mind.

1. You have manually put your changes under default folder instead of use local folders.
2. Your update has contains rm -fr SPLUNK_DIR or something else which has removed those local folders.

r. Ismo