OED Developer Information (detailed)

There are many details and good ideas to use as an OED developer. This page tries to give you some of them.

Faster OED installation

The easiest way to install OED is to use:
docker-compose up
This is equivalent to the full OED install command of
docker-compose run --service-ports --rm web npm run src/scripts/installOED.sh

When you do this it does a complete npm package install by first removing the node_modules/ directory. Downloading all the needed packages for OED can take many minutes. (full install output) You can avoid the package download by using
install_args="--keep_node_modules" docker-compose up
or the full install of
docker-compose run --service-ports --rm web npm run src/scripts/installOED.sh --keep_node_modules
It is very important to note that this skips updating packages. If you change package.json or package-lock.json directly or by pulling in code that does this, you should not use this option but do the full install above after this and then you can use this faster process. If you do not then you will see errors and/or unexpected behavior when OED installs/runs. (skipping NPM install output)

Resetting the Postgres database

During the initial install of OED, Postgres detects that you have not run before for this install and initializes the database system. It keeps the information in the postgres-data/ directory in the main OED directory. If you or a pull changes the database configuration, it will not automatically be changed on the next OED install. When we change versions of OED we use a migration system to upgrade the database but this may not be available during the development cycle. Given this, it is often easiest to force Postgres to reinitialize the database. Note that doing this will wipe out any data (meter, readings, preferences, etc.) that you have for this instance of OED. If you wish to force Postgres to reinitialize then do the following:

  1. Remove the postgres-data/ directory from the main OED directory. While you can delete it, it is probably safer to move the entire directory somewhere else outside your OED directory. That way, if something goes wrong, you can remove the new postgres-data/ directory and put this one back.
  2. Do an install of OED. Remember if package changes were made then you need to reinstall the node_modules. In this case, do not use the keep_node_module option. Note the install will take longer do to the extra initialization.
  3. Sometimes the first attempt at installation will fail when you do this. Try it one or two times more to see if it will work. (We are trying to determine why this happens.) If not, replace the postgres-data/ directory with the one removed and this should get you running but will not give you a new version of the database. If you have this issue then contact us for help (see link at bottom of page).

OED runs in the containerized Docker environment. This means the software and files created are within this container and disappear when you stop running OED (shut the container down). The one exception to this rule is the information stored in the PostgreSQL database. Because this information needs to exist between OED runs to keep the database initialization and OED data, the information is kept within your OED installation in the postgres-data/ directory on your machine's file system. This is why removing the postgres-data/ directory resets the database within the Docker container.

Stopping OED

If you go to the terminal where OED is running, you can type ^-c (control and letter c) together to stop OED. Note it is best to only do this once since a second time causes a force stop. An alternative is to use docker-compose down in a second terminal that is in the same directory as where the install is happening. This can also be useful if you are having an issue with your docker container(s) since it cleans them all up.

Normally, doing ^-z will put the process into the background. This does not work with a running OED.

Seeing what is happening with OED

Sometimes OED will not act as you expect and you want to know what is going on. There are two normal ways to do this:

The warning:
WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
may or may not show up in the output. The complete answer is not known but it appears to be a transient error in PostgreSQL writing log information. It has never caused any problems.

Getting a running OED to recognize code changes

As a developer, you are often changing the code to see what impact it has on a running version of OED. In general, the system should recognize these changes and update the running version. When that happens, in the terminal where you started OED, you see:

web_1       |  94% after seal                                                                 
web_1       | [at-loader] Checking started in a separate process...
95% emitting 
web_1       | [at-loader] Ok, 0.006 sec.

Where the % shown will vary and sometimes it says "seal" instead of "emitting". The update can start almost immediately after a saved file or it can take a modest amount of time. If it does not happen, you can restart the system (forces the update) by typing "rs" in the terminal where the output from OED is showing and you started OED. This is indicated during the install from the line:
[nodemon] to restart at any time, enter `rs`
Note that once the at-loader ok line is seen, you may need to reload your web page to see changes. Whether this is necessary depends on what was changed (normally server vs client code).

OED environment variables

If you want to know about the environment variables that control OED then see the environment page.

OED Scripts

OED has a number of scripts that are run via commands in the package.json file. A complete listing with descriptions is on the NPM script page.

Migrating OED

The script src/scripts/updateOED.sh should update all dependencies and migrate the OED database if changes are needed. This may not work (see ideas above) if migrations are not yet available or this has been done before for the same version of OED.

Having multiple versions of OED

Most developers keep one clone of the OED software. They switch between branches as needed. However, if you are working on multiple versions of OED, esp. where the node modules are changing or the database configuration changes, then you can avoid having to do a full install (see above) by cloning separate copies of OED. When you install it will do the clone you are currently located in (the current directory in your terminal). The database information and all code is kept separate for each installation as part of the Docker containerization.

Rapid testing of a one or a few test(s)

When you make changes to OED or develop new tests, it may be the case that only one or a few tests fails when you run testing (or you want to focus on one test at a time that fails). Running all the tests is much slower than running a single test. Thus, to run a single test, do:
docker-compose run --rm web npm run testsome -- <first test> [<second test>]
where you replace < xxxx test> with the path to the test you want to run and [ ] indicates this is optional and you can repeat as many times as you want for more tests.

For example, to run the web groups test, you would use:
docker-compose run --rm web npm run testsome -- src/server/test/web/groups.js
and to run the web groups test and the db baselineTests use:
docker-compose run --rm web npm run testsome -- src/server/test/web/groups.js src/server/test/db/baselineTests.js

Accessing the database

You can access the Postgres database that OED uses within the docker container. In a second terminal (the first should have OED running), do docker-compose exec database psql -U oed. You will now be in the Postgres command line for the OED database. Note this is the live system so you can make changes that the running application will see (and damage the information too!). It is assumed you know some SQL and Postgres command line commands if you are doing this. One important one is \q which will get you back to the terminal by leaving the Postgres command system.

There may be times you want to backup/restore the Postgres data for OED. You can do it by

Database timeout on testing

Sometimes when you run the OED tests for the first time in a while, you get an error about the database timing out. The message is normally something similar to "Error: Timeout of 15000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves." If this happens, rerun the test command. It normally works the next time (or within a total or three tries). We are working to change our testing and installation so Postgres is ready before operations continue so this problem will go away. The current fix is to allow a test to take up toe 15,000 ms or 15 seconds to complete so Postgres has time to start up and that is why you see "15000ms" in the error message. If the problem persists, you can go into package.json and change the 15000 to be a larger value (such as 30000). This means mocha will not terminate a test until 30 seconds have passed. The only known negative of doing this is that a test that will not complete due to an error will take longer for mocha to terminate. However, this is uncommon in OED tests so the change should be fine.

check permission error

A few developers have reported that when they run docker-compose run --rm web npm run check they get an error similar to "Error: EACCES: permission denied, scandir '/root/.npm/_logs". The only current fix is to run outside of docker by doing npm run check. It is believed that this issue is due to doing installs as root and then not being root when inside docker but this is not certain. However, at least one developer found that changing the file permissions did not fix the issue. If anyone has this issue or finds a solution then please let us know.

Running without the database container

On rare occasions, you may get a database failure during the startup of the database container during the install/startup of OED. (This is not the transient issues noted during the install where the script reports it is continuing). Generally OED will tell you an issue happened in the console output and Docker indicates the database container is not running. In general, the underlying issue should be addressed so you have a correct OED installation. (This happened, for example, during Windows WSL installs until an file permission issue was resolved.) In rare circumstances, you still want to run the OED web container knowing that any operations that touch the database will fail. There is a parameter to the install script that allows this and can be done with:
install_args="---continue_on_db_error" docker-compose up
or the full install script (not using docker-compose up) of
docker-compose run --service-ports --rm web npm run src/scripts/installOED.sh --continue_on_db_error
OED will still try to install the DB, fail and then continue to try to install the web container. You should see the regular messages in the console indicating the web container is running as expected.

TypeScript Wisdom

TypeScript is great, but sometimes it requires some rather strange incantations, especially when working with external libraries. We cannot provide information on everything you need to know but here is one of interest:

Regarding InjectedIntlProps from react-intl

When using react-intl's higher order component functionality, some incantations are required. When defining a component, it's necessary to express that the component's props are actually (your props) ∪ (internationalization injector). This can be done via the & operator, as follows:

interfaceSomeProps {
    prop1: number;

interfaceSomeState {
    state1: number;

class SomeComponent extends React.Component<SomeProps & InjectedIntlProps, SomeState> {
    constructor(props: SomeProps: & InjectedIntlProps) {
        // ...
    // ...

export default injectIntl<SomeProps>(SomeComponent);

If the union type is used often, it might be cleaner to create a type alias:

type SomePropsWithIntl = SomeProps & InjectedIntlProps;