In the previous installment I linked to a plan I'd created that had the first set of tasks about designing a base test system that provides automation for testing. In this one I'll talk a bit about what I did to accomplish that and how to look it over and reproduce my work (run the test successfully).

git-bug issues

I also talked about modifying a couple requirements last time and that I needed to track issues somehow. I did decide to experiment with git-bug and so that's become a part of the process here. Unfortunately, it looks to me like I didn't verify it was usable if I deleted my current working copy because it isn't in the copy I made to verify my test process can be recreated. So before I talk about how to use that system I have to figure out how to fix it and may be changing my mind about using it.

The issue is that my user identity doesn't seem to be available to adopt before I pull and I can't pull without adopting my identity. This is fixed in the version they just released a couple days ago but I haven't updated yet. I've verified you can pull and review the issues if you want and have the latest git-bug:

git bug pull
git bug bug

It's not really going to tell you a whole lot right now though.

test design

The test design document describes how the tests are made to work. The system uses docker for now but I have plans to switch to podman to hopefully eliminate some issues with the setup, which reflect the fact that I'm pretty lazy about the security of my own development docker environment. There are a couple things I really wanted out of this:

  • testing takes place on a faked internet where the real thing can't be accessed.
  • tests can be fully automated including the setup/teardown of different faked internets.

The top diagram of the page describes the components. For now the implementation is just a "hello service" that returns "Hello, World!!!" when you do a get on its root resource.

A simple sort of certificate authority is made from shell scripts. The CA's certificate is registered as authoritative with the service and contact systems. The CA's certificate generation script is used to create keys and certificates for the form and service servers, which are used by them to be accepted by the service and contact systems and then encrypt traffic. I wanted to use ACME here but I'm going to figure that out later.

With the certificates made and the service built, some docker images can be built that have the necessary software and components to make a fake deployment. There's some shell scripts to do this building that are all tied together into /tools/build.sh.

Once that's done, networks can be created that have dnsmask to resolve domain names (docker has its own but this is simulating real names), any number of form systems, and any number of services that can run behind nginx-proxy images. Within this network the contact system can be available or join and then access the form or service to test it.

One example was made in single-form that just has one form system with one service system along with the dns resolver and a contact image that runs firefox such that it can be accessed from a browser on the localhost.

Automation is then added to this by having a testnet isolated network available that a contact system running selenium can join on which can also run another image that has pytest and docker-in-docker to both start up the test network the contact system is in and also control that system's selenium through webdriver.Remote in pytest scripts. An example test was created that starts up the single-form test network and then fetches the service message to validate it. This testnet is created by nix develop on entry but it doesn't then get deleted until the developer does or the system resets.

Why all of this?

It's important to make test automation a first-class citizen in the project plan and design. I've tried here to cover all the basic things I need from the test system. Namely that:

  • I'm able to automate most even if I can't use a free CI/CD server.
  • The system is tested in as close to the production environment as I can mimic.
  • My tests are isolated and aren't erroneously affected by the real world or visa-versa.

By using virtualization I can simuulate a great deal about the production environment in a way that doesn't alter the way the system has to behave. It may be additionally useful or preferable to use a completely virtualized system image but containers can do a lot here and can do enough for now. The original plan I had did hope I'd have an actual base system image I'd start with and then install things on and set up in virtual networks with kvm instead, but this implementation still sufficiently serves the purpose of exercising the design.