Use the extended database scheme in weewx

Are you missing data in your reports?

Or did you get new sensors, and the data won't show up?

If you're using weewx for a longer time or haven't configured it properly, your data might be stored in the "old" default database scheme.

While developing our latest weewx skin, NeoWX Material, we added all available sensors to this skin. Some values are calculated by software, like cloudbase or windchill. Windchill showed up fine, but we couldn't get the cloudbase displayed.

After many searches and a bit of troubleshooting we found the culprit of this strange error.

We were using the default database scheme of weewx. This is fine for most sensors but especially more uncommon sensors or newer values weren't stored in the databse and therefore couldn't be displayed.

With weewx version 4 a new database scheme was introduced.

If you're installing weewx for a new weather station and don't have any old data you just need to use weewx like it's described in the weewx documentation.

As you can see in the weewx.conf file at line 535 (version 4.4.0) the default database scheme is now the extended one. So this will be automatically used.

A small note on the used database system:

Please do the few extra steps and set up a MySQL / MariaDB database for weewx. A real database works much better and gives you more opportunities, especially when you want to store weather data for a long period of time, like we do. SQLite has too many limitations for this.

This is the most common case. You're already having a lot of data in your database, and your weewx runs for a long time already. Probably since version 3 or earlier, which is also the most likely reason for this problem.

Basically you just use a weewx utility which creates a new database with the new extended scheme and then migrates all your data.

We used MySQL for this. If you're currently using a SQLite database, please see the transfer option of wee_database on how to transfer your data from SQLite to MySQL / MariaDB.

|> Step 1: Prepare weewx

First you should update weewx to the latest version.

Then stop weewx. Depending on your system this might be something like systemctl stop weewx.

If you want to create a new, clean weewx.conf you can move your current weewx.conf and run wee_config to create a new one. Then you just need to adjust a few parameters which you can easily copy from your moved weewx.conf.

See the wee_config docs for more information.

In this case the extended scheme should be set automatically.

If you'll keep your weewx.conf open it and scroll to the end. In the default weewx.conf this is line 535.

In the [DataBindings] section you find the [[wx_binding]] sub section where the schema is set at the end.

Update this line to: schema = schemas.wview_extended.schema. This will set the wanted extended database schema.

|> Step 2: Prepare the database

Weewx will create a new database. This will be the name of your current database with a _new suffix.

So in most cases the existing database weewx will be migrated to the new database weewx_new.

Weewx needs the right permissions to work with the new database. Grant the permissions in SQL:

GRANT select, update, create, delete, insert ON weewx_new.* TO weewx@localhost;

Depending on your setup you might need to change the database name weewx_new, the weewx database user weewx or the host. Don't forget to flush the privileges with: FLUSH PRIVILEGES;.

Now weewx can create the new database and do all the work.

|> Step 3: Running wee_database

Weewx provides a very useful tool, called wee_database.

If you installed weewx via a package manager, this command will probably be in your $PATH. In other cases this will mostly be in /home/weewx/bin/wee_database.

With this tool we can easily create the new database and transfer all the data. See this linked documentation above for more information and other options this tool provides.

To do this task we will "reconfigure" the database. As the help says:

--reconfigure  Create a new database using configuration information
               found in the configuration file. In particular, the
               new database will use the unit system found in option
               [StdConvert][target_unit]. The new database will have
               the same name as the old database, with a '_new' on
               the end.

The command will be: wee_database --reconfigure

Depending on your amount of data this can take a while. Thanks to our Davis Vantage Vue the data is updated often. We decided to add a new record every minute, which resulted in nearly 2 million entries. The whole migration it took about 30 minutes on our server.

|> Step 4: Set the new database

Now that all data is migrated we set the new database as our primary one in weewx.

You can just rename the new and old database, or you update your weewx.conf.

To set the database name, go to the [Databases] section, the [[archive_mysql]] sub section and set database_name = weewx_new. This is at line 550.

|> Step 5: Re-generate daily summaries

The new database is missing the daily summaries.

You can easily create them via wee_database by running: wee_database --rebuild-daily.

|> Step 6: Start weewx again

Now you just need to start weewx again. If everything worked weewx will add all missing records to the new database and will then work as before. Just with a bigger database.

|> Step 7: Optionally delete the old database

We always suggest keeping the database for some longer time. In case something went wrong you can easily restore your old data. You just need to adjust your weewx config.

After some time you can free up the disk space and remove the database.

|> Step 8: Use our new material skin

If you want to make use of all your fresh new available data and display it in a beautiful way, please have a look at our latest weewx skin, NeoWX Material.

This skin has many features, including interactive graphs, a large almanac, a separate telemetry page and lots of options. The documentation shows you all the available settings.