Skip to main content

ALKiln automated testing

warning

WIP (Work in progress)

Welcome to the ALKiln testing framework! It helps you check that your docassemble interviews work the way you want them to. ALKiln acts like a user, following your instructions to do things like fill in fields, upload files, sign in, and check your PDFs for accuracy.

Any basic docassemble package can use ALKiln.1 ALKiln (Assembly Line Kiln) took its name from where it was born - the AssemblyLine project - but it can be used with any docassemble project.

🚧 ALKiln is under very active development. Give us feedback and suggest ideas by opening issues at https://github.com/SuffolkLITLab/ALKiln/issues.

Quick start​

Ready to start testing? Need to quickly refresh your memory on some important points? Start here.

See a few more reminders here.

What will you find in these documents?​

ALKiln topics you can explore:

  • Setting up: Learn different ways to install ALKiln for your package or organization and information about those test types
  • Writing tests: What you need to know to effectively write tests
  • Troubleshooting: Help with common errors and warnings
  • Support for older Versions: Guides for using older out-of-date features
  • Frequently Asked Questions (FAQ): Find answers to common questions
  • Security: Find out how to stay safe by using best practices and how we protect you too

What can I test with these tests?​

There are two main purposes for ALKiln tests right now.

  1. You can start using the simple "it runs" test right away. It just makes sure that your interview starts without an error. Even that can be useful. For example, if you forget to commit a template to GitHub, that test will catch it.
  2. The next type of testing comes a while after that, once your interview has stabilized. For example, the variable names are less likely to change. Then you can add more comprehensive ALKiln tests. After that, they will keep an eye on things for you. While you refine your interview by changing wording, the order of fields and pages, adding translations, and so on, the ALKiln tests will let you know if and when your changes break your interview.

Why automate testing?​

This may not be a good sales pitch, but you do not always need automated tests. Automated tests need to be updated along with your code. That may not be worth your time for some projects or at some stages in a project.

That said, the more complex your project gets, the more useful automated tests can be.

When you start creating a project, you test it by hand. You open the interview and run it yourself, clicking through every path your user might take. That does not scale as a project grows. Often, there is no time or energy to test all the paths accurately. You end up having to trust that the ones you skip still work. Automated tests make sure you know when your edits break your code.

Tests can protect your users from bugs that can:

  • Lead them down the wrong path
  • Give them the wrong output in their generated file
  • Stop them completely with an error
  • Lose their trust in you

You can write your simple first test right away if you want. Wait to write more complex tests till your code is more stable. "More stable" means that the variables for the section you are testing are mostly staying the same.

When you add a new required field or change the variable name or values of a user's field, you will need to update those tests.

To hear more about why and when to test, watch the first part of this Docacon 2023 presentation:

What libraries does ALKiln use?​

ALKiln uses cucumber, puppeteerjs, cheerio, and runs the assertions using the mocha and chai libraries.

Even though ALKiln is built using cucumber, this framework has a different, less lofty, purpose. Cucumber focuses on BDD (behavior driven development). This framework mostly deals with regression testing and other conveniences.

For security, ALKiln freezes the versions of its third-party libraries and only updates them after we check their changes.

Where are our repositories?​

How can you contribute?​

ALKiln docs collaborators​

Both humans and LLMs have contributed to this documentation.

Regarding the former, we asked for feedback, ideas, and technical audits from staff at SuffolkLITLab, Recurse Center collaborators, friends, family, and of course you, our users.

As for the latter, we sometimes used the free version of phind to offer sometimes better (and often worse) suggestions for re-wording and phrasing.

Footnotes​

  1. More complex packages might have trouble using ALKiln. There are some interview features ALKiln cannot yet handle. For example, object_multiselect fields. ALKiln also may not understand custom validation code or complex CustomDatatype fields if you make them. If you run into problems, file a clear and descriptive issue on our repository and we will see if we can help. ↩