OED Website
OED Help
Developer Home
Getting Started
There are many details and good ideas to use as an OED developer. This page tries to give you some of them. The topics covered are:
If you are running in a VSC container, then you can simply click the green icon for the Remote Container and select "Close Remote Container". You can also shut the container down manually (by right clicking and choosing "stop") but that is normally only done if something went wrong in shutting down the container in the usual way.
If you are running OED outside of Visual Studio Code then go to the terminal where OED is running, you can type
^-c (control and letter c) together to stop OED. Note it is best to only do this once since a second time causes
a force stop. An alternative is to use docker compose down in a second terminal that is in the same
directory as where the install is happening. This can also be useful if you are having an issue with your docker
container(s) since it cleans them all up. Normally, doing ^-z will put the process into the background. This
does not work with a running OED.
Sometimes OED will not act as you expect and you want to know what is going on. There are two normal ways to do this:
As a developer, you are often changing the code to see what impact it has on a running version of OED. In
general, the system should recognize these changes and update the running version. If you change code under the
client/ directory, the output will include a rebuild of the code. When that happens, the output
will show this line if it worked. This will be in the terminal where OED is running or in nohup.out if you are
working inside VSC.
oed-web-1 | webpack 5.62.0 compiled successfully in 24100 ms
If it fails then you will see one or more failure lines (with failure in red) rather than success. It will also
show that it is done rebuilding when you see the line:
oed-web-1 | \
It is not safe to reload the new version in the web browser until the rebuild finishes. You must reload to see
the new code. If you change code on the server side, you will not see a rebuild but after a few lines of output
you will see:
oed-web-1 | [INFO@2022-04-24T16:31:17.449Z] Listening on port 3000
and then you can see the changes. Since all these changes are on the server, you do not need to reload the web
browser.
Normally the output is seen quickly but it can take a little while for webpack to detect the changes. If it ever fails to do this you could shut down OED and restart it but it is generally much easier/faster to go into some file, add a space, remove the space and save the file. This change (that really does nothing) should cause webpack to start the rebuild.
In several cases, developers on Windows have reported not having the code rebuild on save. It turned out that the code was not on the WSL/Linux partition (generally under /home). As described in WSL on Windows, it is important to do this properly.
You can access the Postgres database that OED uses within the docker container. In the database docker container
(see info on accessing container) terminal do:
psql -U oed. You will now be in the Postgres command line for the OED database. Note this is the
live system so you can make changes that the running application will see (and damage the information too!). It
is assumed you know some SQL and Postgres command line commands if you are doing this. One important one is
\q which will get you back to the terminal by leaving the Postgres command system.
There may be times you want to backup/restore the Postgres data for OED. You can do it by
pg_dump -U oed > dump_$(date +%Y-%m-%d"_"%H_%M_%S.sql)
docker cp /path/to/dump.sql container_name:/dump.sql and then restoring it into the database
with psql -U oed -f /dump.sql. The /path/to/dump.sql is the file with date & time created in
the dump step. TODO Note someone should test/see if you really need to put it into the container first and
this has not been tested since we went to working in the database container. When you make changes to OED or develop new tests, it may be the case that only one or a few tests fail when you
run testing (or you want to focus on one test at a time that fails). Running all the tests is much slower than
running a single test file. Thus, to run a single test, in the web/vsc docker container (see info on accessing container) do:
npm run testsome <first test> [<second test>]
where you replace < xxxx test>
with the path to the test file you want to run and [ ] indicates this is optional and you can repeat as many
times as you want for more tests.
For example, to run the web groups test, you would use:
npm run testsome src/server/test/web/groups.js
and to run the web groups test and the db baselineTests
use:
npm run testsome src/server/test/web/groups.js src/server/test/db/baselineTests.js
If you are using VSC you can easily get the path to any open test file. Right click on the file name in the tab for that editor window and choose "Copy Relative Path". This can then be pasted into the needed command.
During the initial install of OED, Postgres detects that you have not run before for this install and initializes the database system. It keeps the information in the postgres-data/ directory in the main OED directory. If you or a pull changes the database configuration, it will not automatically be changed on the next OED install. When we change versions of OED we use a migration system to upgrade the database but this may not be available during the development cycle. Given this, it is often easiest to force Postgres to reinitialize the database. Note that doing this will wipe out any data (meter, readings, preferences, etc.) that you have for this instance of OED. If you wish to force Postgres to reinitialize then do the following:
rm -rf postgres-data in the main OED directory.
Note you may need to be root to remove this directory. If this is the case, use
sudo rm -rf postgres-data where you will be prompted for the root password.
wsl -u root that will put you in your WSL/Linus
terminal as root. You can then directly delete the postgres-data/ directory. You should be very
careful using this terminal as you can do damage as the root user.docker compose down in a terminal in the main OED directory will also accomplish this.
Otherwise, you are likely to
get the following error when you restart OED:
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c5c261024a2161c37bee3afcd0931bd36812e943519b816556c0bfbd60899ef2" to rootfs at "/var/lib/postgresql/data/pgdata": mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/c5c261024a2161c37bee3afcd0931bd36812e943519b816556c0bfbd60899ef2:/var/lib/postgresql/data/pgdata (via /proc/self/fd/14), flags: 0x5000: no such file or directory: unknown
OED runs in the containerized Docker environment. This means the software and files created are within this container and disappear when you stop running OED (shut the container down). The one exception to this rule is the information stored in the PostgreSQL database. Because this information needs to exist between OED runs to keep the database initialization and OED data, the information is kept within your OED installation in the postgres-data/ directory on your machine's file system. This is why removing the postgres-data/ directory resets the database within the Docker container.
If you want to know about the environment variables that control OED then see the environment page.
OED has a number of scripts that are run via commands in the package.json file. A complete listing with descriptions is on the NPM script page.
The script src/scripts/updateOED.sh should update all dependencies and migrate the OED database if
changes are needed. This may not work (see ideas above) if migrations are not yet available or this has been
done before for the same version of OED.
Most developers keep one clone of the OED software. They switch between branches as needed. However, if you are working on multiple versions of OED, esp. where the node modules are changing or the database configuration changes, then you can avoid having to do a full install (see above) by cloning separate copies of OED. When you install it will do the clone you are currently located in (the current directory in your terminal). The database information and all code is kept separate for each installation as part of the Docker containerization. Note you cannot run multiple version of OED simultaneously unless you change the ports it uses (this is not common).
If a "From" in a Dockerfile is changed, then the version of that software needs to be updated. Docker normally uses its cached version so the update will not happen. To force the update, do:
rebuild container.docker compose up --build where you can add other arguments (such
as keeping the node modules). This is unusual in that it is done in a terminal on your own machine in the
main OED directory and not in an OED Docker container (because you are resetting the Docker containers).
This should make Docker use what is specified in the Dockerfile. If there are no changes then it will not download a new version of the needed software but it will be a little slower for the checks and potentially creating the container. OED tries to announce to the Docker channel for developers whenever a change is make to a Dockerfile that would require this action. This is not commonly done.
On rare occasions, you may get a database failure during the startup of the database container during the
install/startup of OED. (This is not the transient issues noted during the install where the script reports it
is continuing). Generally OED will tell you an issue happened in the console output and Docker indicates the
database container is not running. In general, the underlying issue should be addressed so you have a correct
OED installation. (This happened, for example, during Windows WSL installs until an file permission issue was
resolved.) In rare circumstances, you still want to run the OED web container knowing that any operations that
touch the database will fail. There is a parameter to the install script that allows this and can be done with:
install_args="--continue_on_db_error" docker compose up
or the full install script (not
using docker compose up) of
docker compose run --service-ports --rm web npm run src/scripts/installOED.sh --continue_on_db_error
OED will still try to install the DB, fail and then continue to try to install the web container. You should see
the regular messages in the console indicating the web container is running as expected. If you are working
inside VSC, you will need to edit the src/scripts/installOED.sh script to add the --continue_on_db_error
parameter.
TypeScript is great, but sometimes it requires some rather strange incantations, especially when working with external libraries. We cannot provide information on everything you need to know but here is one of interest:
InjectedIntlProps from react-intlWhen using react-intl's higher order component functionality, some incantations are required. When
defining a component, it's necessary to express that the component's props are actually (your props) ∪
(internationalization injector). This can be done via the & operator, as follows:
interfaceSomeProps {
prop1: number;
}
interfaceSomeState {
state1: number;
}
class SomeComponent extends React.Component<SomeProps & InjectedIntlProps, SomeState> {
constructor(props: SomeProps: & InjectedIntlProps) {
super(props);
// ...
}
// ...
}
export default injectIntl<SomeProps>(SomeComponent);
If the union type is used often, it might be cleaner to create a type alias:
type SomePropsWithIntl = SomeProps & InjectedIntlProps;
Sometimes when you run the OED tests for the first time in a while, you get an error about the database timing out. The message is normally something similar to "Error: Timeout of 15000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves." If this happens, rerun the test command. It normally works the next time (or within a total or three tries). We are working to change our testing and installation so Postgres is ready before operations continue so this problem will go away. The current fix is to allow a test to take up toe 15,000 ms or 15 seconds to complete so Postgres has time to start up and that is why you see "15000ms" in the error message. We have also set the timeout to be longer in a few tests where the issue was happening. If the problem persists, you can go into package.json and change the 15000 to be a larger value (such as 30000). This means mocha will not terminate a test until 30 seconds have passed. The only known negative of doing this is that a test that will not complete due to an error will take longer for mocha to terminate. However, this is uncommon in OED tests so the change should be fine.
A few developers have reported that when they run docker compose run --rm web npm run check they get
an error similar to "Error: EACCES: permission denied, scandir '/root/.npm/_logs". We do not believe this will
happen any longer now that developers are asked to work within the Docker container. For the historical record
the only know fix was to run outside of docker by doing npm run check. It is believed that this
issue is due to doing installs as root and then not being root when inside docker but this is not certain.
However, at least one developer found that changing the file permissions did not fix the issue. If anyone has
this issue or finds a solution then please let us know.
The installation process was updated in late February 2022 so the node modules are not updated unless needed.
This means the node modules will only be updated if the package.json and/or package-lock.json file is update
whether by directed edit or from pulling updating versions. As a result, the old method of
install_args="--keep_node_modules" docker compose up should not be needed if you are running
current versions of OED software. The keep_node_modules parameter was left for backward
compatibility and in the unlikely case where a developer wishes to use it. As expected, the steps and output is
different when there is no node module installation as seen in skipping NPM
install output.