Set up ALKiln tests

warning

WIP (Work in progress)

Welcome to our guide on different ways to set up your ALKiln tests. You can use as many of these methods as you want. You can also disable tests if you need to.

Should I read this?

Maybe not. For example, if you have a GitHub repository and it already has a ".github" folder with a workflow file, you can probably skip this page.

People who only write tests: Skip this and see the documentation on writing tests.
GitHub repository or organization managers or admins: This guide might be useful to you.
Docassemble server managers or admins: This guide might be useful to you.

Types of tests

There are three kinds of tests:

GitHub+You™ - GitHub-triggered tests that create and test a Project on your docassemble server. This type of test runs into the fewest setup barriers, so we recommend this method if you are doing this without help.
ALKilnInThePlayground™ - Tests that you trigger manually from an interview you install on your docassemble server.
GitHub Sandbox™ - GitHub-triggered tests that create their own docassemble server on GitHub and run tests there.¹

Each type of test has its advantages and disadvantages. This documentation explains some of those. First, here are the different ways to set up for the different types of tests.

Recommended setup: GitHub+You™

This is the way we recommend setting up ALKiln tests for a repository for the first time. It is the method that tends to work most smoothly. Every repository that needs to use this kind of test must go through this setup process.

Remember to use a separate server for these types of tests.

The tests run whenever you commit any code to any branch on GitHub. You can read more details and the pros and cons of GitHub+You™ tests below.

You need:

A developer or admin account on your docassemble server
A package with a repository on GitHub
A collaborator or admin account on your package's GitHub repository²

It could help to know about:

Related topics:

What to do:

Use ALKiln's test setup interview to add ALKiln to your repository. You must run this setup interview for every package you want to test on GitHub. This setup interview can take over 30 minutes, especially if you are unfamiliar with GitHub or with docassemble API keys.

When you set up, you can choose to create your first test. We recommend that you do this. Either way, you can add more tests later.

To summarize, the ALKiln tests setup interview can help you do one or more of the following:

Add GitHub secrets to your repository (or organization³)
Make a new branch in your repository
Add new code to that branch, including a new workflow file
Add a new test file, like our example test, to your Project's "sources" folder
Make a pull request with that branch

Once you accept the pull request and merge the code into your repository's main branch, make sure to update your docassemble Projects to use that new code.

If you are ready to start running tests, you can read about how to add our example test, write tests, or trigger and run GitHub+You™ tests on GitHub.

First test

You can add a short test right away that just makes sure your YAML file runs. If you run the setup interview you can choose to create this test at that time. We recommend you do that. If you set up your tests manually instead:

In the Playground of your Project, go to the "Sources" folder.
Add a file that ends in .feature. For example, interviews_load.feature.
In that file, write Feature: at the top, with a description of the general category of tests that will be in that file.
Add a Scenario: for each interview you want to test. The file should look similar to this:

Feature: Interviews load

  Scenario: The 209A loads
    Given I start the interview at "ma_209a_package.yml"

  Scenario: The Plaintiff's Motion to Modify loads
    Given I start the interview at "plaintiffs_motion_to_modify_209a.yml"

You can wait to write more complex tests till your code is more stable. "Stable" means that your question block variable names and values should be staying pretty much the same.

To learn how to do more with tests, you can read more about writing tests and more about test files.

ALKilnInThePlayground™ tests

If you set up ALKilnInThePlayground™ tests, any developer that logs in can run tests on any of their Projects. You only have to do this once. You can read more details and the pros and cons of ALKilnInThePlayground™ tests below.

Remember to use a separate server for these types of tests.

The ALKilnInThePlayground package installs the ALKiln testing framework directly on your server. For security reasons, avoid installing these tests on your production server.

You need:

A developer or admin account on your docassemble server.

It also helps to have a Project with at least one ALKiln test in it, but you can add that later instead.

It helps to know about:

Installing GitHub packages on your docassemble server
Adding interviews to your docassemble server's configuration's dispatch list

What to do:

Sign in to your docassemble server on your own developer account.
Go to your server's "Package Management" page.
Using the GitHub install method, install the ALKilnInThePlayground package from its main branch.
Follow docassemble's instructions to add it to the dispatch list. This will add it to the server's list of interviews. It may look something like this:

dispatch:
  alkiln: docassemble.ALKilnInThePlayground:data/questions/run_alkiln_tests.yml
  # Your other server interviews

Go to your server's list of "Available interviews". The list should have ALKiln's Playground package in it.
Run that interview.
On the first page of that interview, choose to install the latest version of ALKiln by picking the top item in the list. You can choose to do this whenever we make a new version of ALKiln.

If you are ready to start running tests, you can read about how to add our example test, write tests, or continue the above interview to run your tests.

GitHub Sandbox™ tests (advanced)

The GitHub Sandbox™ tests are the most secure because they never interact with your server. ALKiln makes a server on GitHub itself and runs the tests there.

Every repository that needs to use this kind of test must go through this setup process.

This is an advanced method and may not work for everyone. We are happy to help you with it.

The tests run whenever you commit any code to any branch on GitHub. Below, see details and the pros and cons of GitHub Sandbox™ tests.

You need:

A package with a repository on GitHub.
A collaborator or admin account on your package's GitHub repository².

It helps to know about:

What to do:

Sign in to GitHub.
Go to the package's GitHub repository.
Make a new branch in the repository. The name is unimportant. We often call it alkiln_tests.
Navigate to your package's "sources" folder. These are the folders to get there: docassemble/<YourPackageName>/data/sources.
Add a new file on GitHub. This will be your first test.
Fill in the field for the name of the file. You might call it interviews_load.feature, but it can be anything that ends in .feature.
Copy our example of a first test into that file.
Commit this file to GitHub.
Go back to the main page of the repository. Make sure you are on the correct branch.
Add a new file here too.
In the field for the file's name, enter this text: .github/workflows/a_filename.yml. You can replace "a_filename" with whatever name you want as long as it ends with .yml.
This is your workflow file. Copy ALKiln's own workflow file. Read the notes in that file to see what you need to change. You should pay attention to the notes that start with #### Developer note and follow those instructions.
Commit this file to GitHub.
Go to the repository's "Actions" page to see the test running.
If the tests pass, make a pull request with that branch. If not, troubleshoot the tests.

If you are ready to start running tests, you can read about how to add our example test, write tests, or trigger and run GitHub Sandbox™ tests on GitHub.

GitHub+You™ test details

Any developer will trigger GitHub+You™ tests when they commit code to GitHub. They then create and test a Project on your docassemble testing server. They are our recommended test method because they are the easiest to set up.

Remember to use a testing server for these types of tests.

What is ALKiln doing?

When GitHub+You™ tests get triggered, ALKiln does the following:

ALKiln creates a new Project for the testing account you set up on your server. If you logged into that account at the right time, you would be able to see the new Project in the list of projects.
It uploads the relevant GitHub branch of your package into the Project.
For each test, ALKiln pretends to be a human and goes to the url of that Project's interview (whatever interview you named in the test).
It tries to fill out the fields and take the actions the tests describe.
When all the tests are done, ALKiln deletes the Project it created.

The GitHub+You™ tests need to create a Project on your testing server and pull code into it. Because of that, you need to create a developer account for these types of tests.

GitHub can show you information about your running tests. When the tests are done, you can see the "artifacts" of the tests.

Pros

ALKiln on GitHub will test any changes any team member commits. In ALKilnInThePlayground™ tests, you only test your changes.
On GitHub, there are ways to schedule tests to run regularly - every week, every day, or whatever schedule you prefer. If packages you depend on change and break your code, those tests will let you know.
If you've forgotten to add a file to the package that an interview depends on (like a missing template file), GitHub will catch that. It installs the package freshly in a new Project, so it only has the files that have been committed properly.
Unlike ALKilnInThePlayground™ tests, you can continue editing most of the files in the same Project while the tests run because the GitHub+You™ tests create a whole new Project of their own to test with. The only files you must avoid editing are Python files because saving them reloads the server, causing problems for tests.
You can configure your GitHub "action" to do other things, like create GitHub issues whenever tests fail.

Cons

For everyone:

bookmark

The setup for GitHub+You™ tests is more reliable, but also more complex than the other methods. The ALKiln test setup interview tries to help with that and even people with developer access can do it, but it is complex in its own ways⁴.
When GitHub+You™ tests start up, they pull your package code into your docassemble server, just like any other developer. This means that if you test a package with a python module, your docassemble server will reload, slowing down access for other people who are using the server.
Your tests might be more flaky. That means they may fail for reasons that don't have to do with the interviews. For example, if your tests are running while your server is reloading, ALKiln will keep running, so other tests will still have an opportunity to continue, but individual tests may fail even if your code is correct. Your server can reload for many reasons - updating your config, installing a new package, another GitHub+You™ test pulling in a module when it is setting up, etc. When tests fail meaninglessly, it slows down your development cycle and sometimes people end up turning off the tests instead of dealing with the complexity.

For some people:

If you have restrict input variables set to True in your server's config and have signature screens in the interview you are testing, you must have the trigger variable HTML in your interview for signatures.

ALKilnInThePlayground™ test details

The ALKilnInThePlayground™ tests are automated tests that you trigger manually from an interview you install on your docassemble testing server. The tests then run right on that server.

Remember to use a testing or development server for these types of tests. You can read more about running ALKilnInThePlayground™ tests in other documentation.

The ALKilnInThePlayground package installs the ALKiln testing framework directly on your server. For security reasons, avoid using these tests on your production server.

What is ALKiln doing?

When you run the ALKilnInThePlayground™ tests while on your docassemble testing server, ALKiln does the following:

For each test, ALKiln pretends to be a human and goes to the url of the Project and interview you specified.
It tries to fill out the fields and take the actions the tests describe.

When the tests are done, the interview will go to the next page where you can see the "artifacts" of the tests.

Pros

You can skip putting your package on GitHub.
Running ALKiln on your server lets you loop through your development cycle faster - edit, test, edit, test. You don't have to go through committing your code to GitHub every time.
You can more easily limit your tests to a specific set of tests using tag expressions. You can use tags on GitHub too, it just takes more steps.
The full report output is easier to get right away.
If you also have GitHub tests, running ALKilnInThePlayground™ tests before you commit can often help you avoid the red "x" that appears next to failing tests on GitHub. The red "x" is a useful piece of information, but it can be uncomfortable.

Cons

For everyone:

You cannot edit the Project that the test is testing.
You will miss catching some kinds of mistakes. For example, ALKilnInThePlayground™ tests will miss "File not found" errors.
Your tests can be more flaky. For example, if your server reloads while the tests are running, the tests may get unexpected results. You may have to start the tests all over again.

For some people:

If your interview uses docassemble background actions then your server must let at least 3 celery processes run at the same time. You can manually set the number of celery processes in your server's config file.
If you are using a docassemble Docker image created before April 9th, 2023, you will have to do a "system upgrade" to update your server to a newer docassemble Docker image. Any version of https://github.com/jhpyle/docassemble-os that is older than 1.0.8 uses versions of node that are too old for ALKiln. As of February 11th, 2024, ALKiln is using node version 18. A lot of people have upgraded already and it is a very good idea to do that, for security at the very least. If you do need to upgrade, though, you need to back up all your data, delete your old image and container, get the new image, create a new container, and restore your data. It may sound simple, but it is a tricky process.
If you have restrict input variables set to True in your server's config and have signature screens in the interview you are testing, you must have the trigger variable HTML in your interview for signatures.

GitHub Sandbox™ test details

Any developer will trigger GitHub Sandbox™ tests when they commit code to GitHub. They create their own docassemble server on GitHub and run tests there.

The GitHub Sandbox™ tests are the most secure because they never interact with your server and the isolated server they each create is temporary. Unfortunately, these tests also sometimes have more setup complications.

What is ALKiln doing?

When you run GitHub Sandbox™ tests, ALKiln does the following:

ALKiln creates a server on GitHub itself
It installs the relevant branch of your package on that server
For each test, ALKiln pretends to be a human and goes to the url of the interview named in the test
It tries to fill out the fields and take the actions the tests describe
When all the tests are done, GitHub deletes the server

GitHub can show you information about your running tests. When the tests are done, you can see the "artifacts" of the tests.

Pros

All the same advantages as regular GitHub tests as well as some more.
This is the most secure of all the methods. The tests don't interact with your server at all.
You can edit whatever files you want on your own server.
In some cases, no GitHub secrets are necessary.
Your tests will generally be less flaky. When a test fails it is more likely to be a problem with your code instead of a problem with the server, giving you better information. That's because the server is isolated, or "sandboxed"¹, on GitHub and no one is going to make that sandboxed server reload in the middle of the tests.
You do not have to set up or update any docassemble API keys or GitHub tokens.
It might be useful to test custom data types on a sandboxed server. A docassemble CustomDatatype affects a whole server. Testing it on a sandboxed server means it cannot affect the CustomDatatypes that are already on your server.

Cons

For everyone:

Tests take about 5-10 minutes longer since they have to do things like create a docker container and start a server.
The GitHub server installs the latest version of docassemble, which might be a different version than the one you use on your production server. Any server other than your production server will be different in some ways, though. Since you should never run these tests on your production server, there is no way to avoid that.
In most cases, you should avoid writing tests that use hard-coded urls that go to your server. That is, avoid using links in your tests that start with your server's address. For example, https://my-server.com. Avoid linking to interviews or other pages there. That will send the tests to your server, which makes them less secure again.

For some people:

Right now, GitHub Sandbox™ tests are unable to handle data stored externally. For example, data stored in an S3 bucket.⁵
If you are including any other packages in your interview, you must make sure you have properly required them in the "Dependencies" section of your Playground Packages page. This is an example of code that uses the Assembly Line package:
```
include:
  - docassemble.AssemblyLine:al_package.yml
```
Because of that code, the author needs to require the AssemblyLine package as a dependency. This is what it looks like to require the Assembly Line package:

Save your choices and commit them to GitHub.
All the packages your package uses, like Assembly Line, must be published on pypi. That is the only place that the sandboxed server can get them.
Those packages must also correctly require all their dependencies and those must be published on pypi as well.
If your package is too big, you will be unable to use this type of test. GitHub Sandbox™ tests install your package on the server itself. That makes sure your package's dependencies get automatically installed too. The method ALKiln needs to use to install your package on the server will reject big packages. We are unsure about the exact limits, but we know it must be less than 30 MB.
If you use any API keys for third-party services, like Google geolocation or recaptcha3, you must write code that has alternatives to using those services. This is fairly complicated. For a simple example with bad code, for recaptcha3 this code might technically work for some situations:
```
code: |
  if user_info().id != 1:
    use_recaptcha
```

Disabling

ALKiln works hard to protect you from security vulnerabilities and there are lots of ways for you to improve your own security. Even so, if you ever become uncomfortable or concerned about your ALKiln tests showing personal or security-critical data, there are multiple ways you can disable the tests.

If you are worried you have already shown sensitive information in your GitHub tests, you can delete the GitHub action that ran the workflow. You can also just delete your GitHub logs and delete your GitHub artifacts.

Disable a test or test file

This is for: Everyone

You can disable one or more tests in a test file.

You can fully disable any test by commenting out its code. Put a # at the start of every line of the test. You can do this to every line in a file if you want. You can do this by editing the file in your Playground or editing it in GitHub.

If you comment out a test, commenting out all its lines is better than just commenting out some. That will avoid messing up other tests.

If you are more comfortable completely deleting a test, that also works. You can also delete the whole file. If your package has a GitHub repository, you must delete the file in GitHub itself.

The old tests will still be in the version history of your repository if you ever want to get them back, but the tests will no longer run or affect your package.

Disable repository tests

This is for: any packages that run tests on GitHub

GitHub lets you disable or limit all tests, or all actions, in your repository. You can read GitHub's documentation for managing repository actions for more information.

Another option is to disable an individual workflow (action). Your package's GitHub workflow file is what triggers ALKiln tests on GitHub.

You can also delete the workflow file that is in your repository completely.

Disabling tests for the whole organization

This is for: any packages that run tests on GitHub

In GitHub, you can disable any workflows (actions) for a whole organization. That includes ALKiln tests.

"The isolation metaphor is taken from the idea of children who do not play well together, so each is given [their] own sandbox to play in alone." - Wikipedia, Sandbox_(computer_security) ↩ ↩²
That is, you need to have the ability to edit the code in your package's GitHub repository. If you own the repository or organization, you are an admin and have the correct access level. If someone else is the owner and you are unsure of your access level, ask an admin what permission level you have for the repository. They can check the repository's collaborators settings. ↩ ↩²
Do you have more than one repository? Do they all belong to a GitHub organization? If so, you can get one of the organization's admins to run the ALKiln test setup interview and create organization secrets. They can also create them manually. Each repository the organization has will use those secrets. You will not have to bother adding them to every repository. Otherwise, you have to create the same secrets for each repository. If one of the values changes, you will have to update it in every repository. ↩
It is possible for you to set up GitHub+You™ tests manually on your own. You must understand the process already. If you do this by hand, you must also have admin access to the GitHub repository. ↩
If you need to use external data and still want GitHub Sandbox™ tests, it is probably possible. You can write your own GitHub action to create a server on GitHub with the right Docker build environment variables, credentials, and config details. You are welcome to use our GitHub action code as a starting point, of course. It is a feature we are hoping to add, but it is currently low on the list. If you do try to do this, we would love to help and to learn from what you do. ↩

Should I read this?​

Types of tests​

Recommended setup: GitHub+You™​

First test​

ALKilnInThePlayground™ tests​

GitHub Sandbox™ tests (advanced)​

GitHub+You™ test details​

What is ALKiln doing?​

Pros​

Cons​

ALKilnInThePlayground™ test details​

What is ALKiln doing?​

Pros​

Cons​

GitHub Sandbox™ test details​

What is ALKiln doing?​

Pros​

Cons​

Disabling​

Disable a test or test file​

Disable repository tests​

Disabling tests for the whole organization​

Footnotes​

Should I read this?

Types of tests

Recommended setup: GitHub+You™

First test

ALKilnInThePlayground™ tests

GitHub Sandbox™ tests (advanced)

GitHub+You™ test details

What is ALKiln doing?

Pros

Cons

ALKilnInThePlayground™ test details

What is ALKiln doing?

Pros

Cons

GitHub Sandbox™ test details

What is ALKiln doing?

Pros

Cons

Disabling

Disable a test or test file

Disable repository tests

Disabling tests for the whole organization

Footnotes