Writing ALKiln tests
WIP (Work in progress)
This page describes your tests files and how to write tests and other useful information. To see the table of contents on a small screen, tap on the dropdown menu at the top of the page. To see it on a wider screen, see the column on the right-hand side of the screen.
Do I need to read this?
If you are writing tests or need to learn about test files, this page is for you. It is easier to write the tests if you have access to the interview YAML code and understand a bit about docassemble.
Quick refresher
If you are coming back and just need some resource reminders, here is a short list:
- You can see what your first test might look like.
- You can generate a first draft for a more complex test with the help of the ALKiln Story Table Step generator.
- If your interview is using generic objects or index variables, you must include special HTML in your interview.
- Every time you change or add a required field variable, you must update the tests that need that variable.
- You write and edit
.feature
test files in your package's Sources folder. - You can check the syntax of your test file by using an editor like the editor at AssertThat.
- If you are using GitHub+You™ or GitHub Sandbox™ tests, those tests will run when anyone commits to GitHub.
- By default, each Step or page must be completed in at most 30 seconds. You can change that with the "max seconds" Step.
- ALKiln creates test reports and pictures. For example, ALKiln saves pictures and the HTML of pages that caused errors. In GitHub, those files are in the zip file in your workflow's artifact section.
- There are some field types that ALKiln cannot yet handle, including some
object
-type fields. - The tests use the gherkin language and syntax.
Here is an example of a more complex test for a quick refresher on some of ALKiln's features:
@children
Feature: I have children
@fast @child_benefits
Scenario: child has benefits
Given I start the interview at "some_file_name.yml"
And I get to the question id "benefits" with this data:
| var | value |
| children.there_are_any | True |
| children.target_number | 2 |
| children[0].name.first | Cashmere |
| children[0].name.last | Davis |
| children[1].name.first | Casey |
| children[1].name.last | Davis |
When I set the var "benefits['SSI']" to "True"
And I tap to continue
Then the question id should be "download"
And I download "some_motion.pdf"
Writing test Steps overview
Most of the code in test files is code for ALKiln Steps.
ALKiln Steps are how you give your instructions to the ALKiln robot. Steps are made of commands1 that ALKiln has created for you. Those commands often need information from you. They are like fill-in-the-blank sentences. A sort of madlibs for your ALKiln robot. For example:
And I set the variable "likes_bears" to "True"
This Step is made of three parts.
And
is one of the keywords that can start a StepI set the variable "___" to "___"
is an ALKiln command with blanks that you need to fill inlikes_bears
andTrue
are the information you put in the blanks so that the ALKiln robot knows what answer to give to a question
That command description can also be written as I set the variable "<variable name>" to "<value>"
. This tells you what information you should put in each of these blanks.
In this case there are two blank spaces you should fill. In the first blank, you should put the name of the variable that the field will set. In the second blank, you should put the value that you want the variable to have.
Each of your tests - a Scenario
or a Scenario Outline
- usually contains many Steps. Any Step can be used as many times as you want anywhere in a test. If you follow good code-writing practices, most Steps work in any human language, so you can tests translations of your interviews.
Steps to start tests
Authors often use these Steps to start their tests, though these Steps can actually be used anytime in a test.
I start the interview at "<file name>"
Use the "start interview" Step to start a new session of an interview.
The Step format:
Given I start the interview at "<YAML file name>"
The blank to fill in here is <YAML file name>
- the name of the YAML file that starts the interview.
Example with just the file name:
Given I start the interview at "favorite_bears"
Example with the file extension:
Given I start the interview at "favorite_bears.yml"
Example with url arguments (url_args):
Given I start the interview at "favorite_bears.yml&user_id=123&from=theinterwebs"
You can also instead use the full url of the interview, though we recommend you avoid that. For example, one advantage to using the file name instead of the full url is that you can then still use the GitHub Sandbox™ test method to run those tests on GitHub.
You must include the interview
Step in each Scenario before setting any fields.
I sign in with the email "<username env var>" and the password "<password env var>"
Use the "sign in" Step to sign into your docassemble server before going to an interview with the "start interview" Step. If you run your tests on GitHub, remember to use GitHub secrets to store this sensitive information.
The Step format:
Given I sign in with the email "<username env var>" and the password "<password env var>"
The two blanks to fill in are <username env var>
and <password env var>
. Each of them must be the name of an environment variable that you have already added to your workflow file.
The <username env var>
variable should contain the username of a dedicated account on your docassemble server. The <password env var>
variable should contain the password of that account.
To use this Step:
- When you create a user for ALKiln on your docassemble server, note the email address and the password for that account.
- Create GitHub secrets or config values to store the email address and password.
Example Step:
Given I sign in with the email "ALKILN_USER_EMAIL" and the password "ALKILN_USER_PASSWORD"
"ALKILN_USER_EMAIL"
and "ALKILN_USER_PASSWORD"
are just examples of environment variable names. You can use any names you want.
The relevant section of your workflow file would look like this:
env:
ALKILN_USER_EMAIL: "${{ secrets.ALKILN_USER_EMAIL }}"
ALKILN_USER_PASSWORD: "${{ secrets.ALKILN_USER_PASSWORD }}"
You can have as many of these accounts as you want.
If you run this in a GitHub+You™ test, you must use GitHub secrets to set the environment variables in order to keep the information secure. You should also use a separate account for signing in like this. Never use a real person's account or information.
ALKiln does avoid taking pictures or downloading the HTML for this Step, even when the test has an error on that page, so that information is still protected. ALKiln also avoids printing the value of these variables anywhere in the report or in the console. Even so, it is possible for you to expose this information.
I sign in with the email "<username env var>", the password "<password env var>", and the API key <api key env var>
Use the "sign in and clean up" Step to sign into your docassemble server before going to an interview with the "start interview" Step, then delete the interview from the account's "My interviews" list. If you run your tests on GitHub, remember to use GitHub secrets to store this sensitive information.
This is very similar to the "sign in" Step above. See that for basic information. The difference here is that you can also make sure that ALKiln deletes the interviews this user creates.
The new blank to fill in is <api key env var>
. It is the name of the environment variable containing an API key you create for the account that signs in.
Example Step:
Given I sign in with the email "ALKILN_USER_EMAIL" and the password "ALKILN_USER_PASSWORD", and the API key "ALKILN_USER_API_KEY"
"ALKILN_USER_EMAIL"
, "ALKILN_USER_PASSWORD"
, and "ALKILN_USER_API_KEY"
are just examples of environment variable names. You can use any names you want. See the "sign in" Step for more details.
To set up the API key, do the following:
-
Ask an admin on your docassemble server to create an API key for themselves in their user profile2.
-
Use that admin API key to create an API key for your user by using the docassemble API endpoint. If you lose this API key, you can make a new one. You can do that in your terminal or command prompt with this code:
$(curl -X POST \
<your server url>/api/user/<user's email>/api \
-H 'Content-Type: application/json' \
-d '{
"key": "<your admin API key>",
"name": "alkiln_key",
"method": "none"
}')<your server url>
is the url of your server in this format: https://your-server-url.org.<user's email>
is the email of ALKiln's user account you name in your sign in Step.<your admin API key>
is the API key your admin made. -
Press enter.
-
Copy the result you get from that curl request.
-
Int GitHub, put that variable in a GitHub secret. Avoid including the outer quotation marks.
-
Add that secret to your environment variables. Here's a GitHub workflow file example:
env:
ALKILN_USER_EMAIL: "${{ secrets.ALKILN_USER_EMAIL }}"
ALKILN_USER_PASSWORD: "${{ secrets.ALKILN_USER_PASSWORD }}"
ALKILN_USER_API_KEY: "${{ secrets.ALKILN_USER_API_KEY }}" -
Write this "sign in and clean up" Step in your
.feature
test file.
If you run this in a GitHub+You™ test, you must use GitHub secrets to set the environment variables in order to keep the information secure. You should also use a separate account for signing in like this. Never use a real person's account or information.
ALKiln does avoid taking pictures or downloading the HTML for this Step, even when the test has an error on that page, so that information is still protected. ALKiln also avoids printing the value of these variables anywhere in the report or in the console. Even so, it is possible for you to expose this information.
I go to "<url>"
Use the "go anywhere" Step to go to any arbitrary url.
The Step format:
Given I go to "<url>"
The blank to fill in here is <url>
- any valid url.
Example:
Given I go to "https://retractionwatch.com"
Test configuration Steps
These Steps can change your settings for your tests. For example, the maximum amount of time one Step or page can take to finish.
the maximum seconds for each Step is <number>
Use the "max seconds" Step to set the maximum amount of time one Step can take to finish before it will fail. This is also the maximum amount of time ALKiln will spend trying to fill out the fields on a page before the test fails.
The default maximum time is 30 seconds.
The Step format:
Given the maximum seconds for each Step is <number>
The blank to fill in is <number>
- the number of seconds to allow a Step to run, or a page to be filled, before the test fails.
Example:
Given the maximum seconds for each Step is 200
You can set the max seconds as your first Step to give a slow-loading interview more time to load. If you are worried about increasing the time too much, you can immediately decrease the time after the interview loads. For example:
Given the maximum seconds for each Step is 200
And I start the interview at "favorite_bears.yml"
And the maximum seconds for each Step is 30
You can also increase this setting before you try to download a large file. ALKiln spends twice the value of this setting to download a file. You can read more about the timing in the "download" Step.
You can use this Step to change the maximum seconds setting at any time in your test.
Increasing the maximum time can have down-sides. Some kinds of failures take the maximum amount of time possible. Since each failed test is re-run, those failing tests will take twice the amount of time that you set for your maximum Step time.
Sometimes authors set the maximum seconds to a shorter time for most of their Steps and only lengthen the time for Steps that need more time. This can speed up failing tests.
I check all pages for accessibility issues
Use the "all accessible" Step to turn accessibility checking on.
Example:
Then I check all pages for accessibility issues
After this Step, ALKiln will check every following page for accessibility issues until the end of the test.
If there are problems with the accessibility of the page, ALKiln will continue to the end of the test, but it will save a separate JSON accessibility report file for a page that fails this check. For example, if text is too small or if contrast between the text and its background is too low. If ALKiln found problems, the test will fail at the end.
ALKiln uses aXe-core to check for problems. You can check the documentation there to understand the failure data better. A human can do a much more thorough job of improving accessibility, but this can be a start.
This is different than the "accessibility" Step, which only checks the current page.
Interactive Steps
You can use some Steps to change what is on the page by filling in fields, pressing buttons or tabs, or performing other actions.
The Story Table Step
Use the "Story Table" Step to set values of multiple fields on multiple pages. It is the Step many of our authors work with the most. In our opinion, it is the most flexible way to fill in fields in your forms, unlike the linear "I set" Step. This makes a test easier to maintain.
The Story Table Step is made of a sentence followed by a table. The data in the table is a snapshot of the user who is filling out the form. Each table row describes one fact about the user - one variable in the interview. It has this format:
And I get to the question id "<target question id>" with this data:
| var | value |
| <variable name> | <value to set> |
| <variable name> | <value to set> |
As you can see, there are three types of blanks you need to fill in for this Step:
<target question id>
is theid
3 of thequestion
block where you want ALKiln to stop using this table.<variable name>
is the name of an interview variable for a form field. Each table row needs one of these in the first column.<value to set>
is the value to set for this row's variable name.
The table must include all the variables and values necessary to get a user from the current page to the page with the given question id. The table must have two columns with the headings var
and value
.
An example table:
And I get to the question id "brought scissors" with this data:
| var | value |
| last_haircut_date | today - 730 |
| wants_virtual_haircut | True |
| user_name | Jaime |
| intro_screen_accepts_terms | True |
This Step is flexible, which makes the test easier to maintain. The order of the rows is unimportant. That means you can edit the order of your pages or move fields to different pages without worrying about breaking the test.
ALKiln sets a table variable every time it finds that variable on a page, no matter how many times that variable appears in your interview.
Avoid using .there_is_another
in your Story Table. Read about using .target_number
instead.
Things to note
You can use the Story Table generator to generate a first draft of a full test file.
If your interview uses generic objects (x
) or index variables (i
, j
, k
, etc), you must have special HTML in your interview to set the values of variables.
The Story Table does not care about extra rows that never get used. This can make Story Tables easier to write, but it also means that the tests will technically "pass" even if variables that should get set are never filled in.
That can be a sign that a question that the user should have seen got skipped or that there is a typo or mistake in the test.
Because of this, the ALKiln report shows a list of unused rows for each table. You can check the report of passing tests to make sure none of the important rows got skipped.
How this works, for the curious folks out there
On each interview page, ALKiln:
- Checks if the page has the
<target question id>
you gave it at the start - If the page does have the
<target question id>
it stops the Step - Otherwise, it looks at each
<variable name>
in thevar
column to see if it can find that name in any fields - For each
<variable name>
it finds, it tries to answer the field using the matching<value to set>
- On pages with
show if
fields, it waits to see if a new field appears - If one does appear, it looks through each
<variable name>
again - It sets any value it can and repeats this process
- When no new fields get revealed, ALKiln tries to continue to the next page
- If it is unable to continue it logs an error and stops
- Otherwise, it repeats the above
ALKiln sets a table variable every time it finds that variable on any page, no matter how many times that variable appears in your interview.
I set the variable "<variable name>" to "<value to set>"
Use the "I set" Step to fill in one value on one page.
If your interview uses generic objects (x
) or index variables (i
, j
, k
, etc), you must have special HTML in your interview to set the values of variables.
The format of the Step:
And I set the variable "<variable name>" to "<value>"
There are two types of blanks you need to fill for this Step. The first set of quotes in this Step contain the name of the variable you need to set. The second set of quotes contain the value you want to set. This is the same as the Story Table's's var
column and value
column.
This Step is "linear". That is, unlike Story Table rows, these "linear" variable-setting Steps have to be in the correct order and you must make sure your test has reached the exact page that has the field. Otherwise your test will fail.
Example fields:
code: |
users[0].favorite_bear.name.first
users[0].favorite_bear.type
---
question: Name of your favorite bear
field:
- Bear name: users[i].favorite_bear.name.first
- Why is this one your favorite?: users[i].favorite_bear.why
input type: area
---
question: Favorite bear type
field:
- Pick one: users[i].favorite_bear.type
datatype: radio
choices:
- Black bear: wild_bear
- Teddy bear: cuddly_bear
- Jaime: maybe_not_a_bear
Example linear "I set" Step:
And I set the variable "users[0].favorite_bear.name.first" to "Ultraman"
And I set the variable "users[0].favorite_bear.why" to "Why not?"
And I tap to continue
And I set the variable "users[0].favorite_bear.type" to "cuddly_bear"
You can use this Step to upload files.
I set the variable "<variable name>" to the GitHub secret "<environment variable name>"
Use the "secret value" Step to use an environment variable to fill in one value on one page.
Even though the name has "GitHub" and "secret" in it, it can also use environment variables. It is still the only way to use sensitive information in your tests. To do that, you can use environment variables with GitHub secrets.
If your interview uses generic objects (x
) or index variables (i
, j
, k
, etc), you must have special HTML in your interview to set the values of variables.
The format of the Step:
And I set the variable "<variable name>" to the GitHub secret "<environment variable name>"
Here, <variable name>
is the name of the variable you want to set and <environment variable name>
is the name of the environment variable that has the value.
Example:
I set the variable "user1_account_password" to the GitHub secret "ALKILN_USER1_PASSWORD"
A lot of authors use this Step to set answers with sensitive information, like an account number. This is because ALKiln is extra careful with the security of these values.
Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables. It even avoids taking pictures when the test has an error on that page and during the sign-in Step. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.
This Step is unable to sign into your docassemble server. You must use the "sign in" Step to sign in to your docassemble server.
I tap to continue
Use the "continue" Step to tap the button to continue to the next page. If the continue button sets a variable, this Step will set that variable to True
. Story Table Steps handle continuing by themselves.
Example:
When I tap to continue
The test will still continue to the next Step if tapping the continue button fails to go to the next page. Because of this, you can then use the "invalid" Step to write tests that check that your answer validation works correctly. However, if the continue button was supposed to work and the next Step tries to interact with the next page, the test will probably fail.
I set the name of "<variable name>" to "<name>"
The "name" Step is specifically for the Document Assembly Line 4-part name questions.
Avoid punctuation. We recommend you just use 2 names - the first name and last name - but you can have all these formats:
- Firstname Lastname
- Firstname Middlename Lastname
- Firstname Middlename Lastname Suffix (where suffix is one of the dropdown suffix choices, like
II
)
And I set the name of "users[0]" to "Devon Upton"
I set the address of "<variable name>" to "<US address>
The "address" Step is specifically for the Document Assembly Line 4-part address questions. Right now, the "address" Step can only handle the format of a US address.
When I set the address of "users[0]" to "120 Tremont Street, Unit 1, Boston, MA 02108"
Remember the commas.
I download "<file name>"
Use the "download" Step to download files so that humans can check that they are correct. After all the tests are done the files will be the artifacts folder of the Scenario (test).
Then I download "file-name.pdf"
Use just the name of the file.
If the file is a PDF file, you can use the "compare files" Step to check that the PDF is correct.
Currently, ALKiln is unable to tell when a file has actually finished downloading, so it runs the download Step for about twice the maximum seconds allowed for a Step no matter what. If a download takes longer than the current maximum seconds value, the test will not fully download the file4.
If you think downloading the file will take more than 60 seconds, use the "max seconds" Step to give the file more time to download. The "download" Step will use the the full amount of time that you give it.
I tap the "<selector>" element and go to a new page
It could help to know about:
The Step:
Use one of the "tap and navigate" Steps to tap an element on the page, like a button, to go to a new page. If you just want to tap the continue button use the "continue" Step.
The format for variation 1:
And I tap the "<selector>" element and go to a new page
The format for variation 2:
And I tap the "<selector>" element and wait <number> seconds
One blank to fill in for these commands is <selector>
, a CSS selector of the element you need to tap. It can be any valid CSS Selector.
For variation 2, the second blank to fill in is <number>
- the number of seconds you want ALKiln to wait after tapping the element.
Example of variation 1:
When I tap the "#awesome_page_link" element and go to a new page
Example of variation 2:
And I tap the ".link-container #even_better_page a" element and wait 45 seconds
This is the only way to explicitly wait after navigating to the new page. Some people try to use the "wait" Step, but that will fail. Alternatively, you can use the "max seconds" Step to give your test more time to load new pages in general.
I tap the "<selector>" element and stay on the same page
It could help to know about:
The Step:
Use the "tap and stay" Step to tap on HTML elements without navigating to a new page. For example, you can tap on collapsible content to show the text inside.
The Step format:
And I tap the "<selector>" element and stay on the same page
The blank to fill in for this command is <selector>
, a CSS selector of the element you need to tap. It can be any valid CSS Selector.
Example:
When I tap the "#an_id .al_toggle" element and stay on the same page
It is a good idea to add a wait
Step after this Step to let the contents become visible before taking other actions, like pressing on other elements or checking for a phrase. For example:
When I tap the "#an_id .al_toggle" element and stay on the same page
And I wait .7 seconds
I tap the "<tab id>" tab
It could help to know about:
The Step:
Use the "tap tab" Step to interact with AssemblyLine's ALToolbox tabs. ALKiln will tap on the tab and then wait until the tab contents are fully visible.
The Step format:
When I tap the "<tab html id>" tab
The blank to fill in here is <tab html id>
- the text of the HTML id attribute of the tab.
<tab html id>
is not a selector. Unlike the other "tap" Steps, leave out the #
symbol (like the example shows).
Example:
When I tap the "TabGroup-specific_tab_name-tab" tab
I wait for <number> seconds
You can use the "wait" Step after the previous Step has completely finished to tell ALKiln to wait before going to the next Step.
In most cases, this Step is unable to help the test wait longer for a page to finish loading. To give a page time to load, use the "max seconds" Step.
The Step format:
When I wait <number> seconds
The blank to fill in here is <number>
- the number of seconds to wait. If you want, the number can be less than a second.
Examples:
When I wait .6 seconds
When I wait 150 seconds
When to use the "wait" Step:
If a test sometimes fails because ALKiln says it was unable find a field or element on the page, and other times the test passes, this Step can help. Use "wait" to pause for a moment before trying to interact. This is usually because some elements can take different amounts of time to fully appear.
Use "wait" to let page elements finish moving around. For example, after you use the "tap and stay" Step to expand content. When elements are moving on the page ALKiln has trouble interacting with them.
Use "wait" to wait for a background process to finish - ALKiln is currently unable to detect a background process finishing.
Use "wait" to give external data more time to load.
Setting values
There are different ways to fill in fields. One is the Story Table Step another is the "I set" Step. Each has different syntax, but they set values the same way.
If your interview uses generic objects (x
) or index variables (i
, j
, k
, etc), you must have special HTML in your interview to set the values of variables.
Be sure to use the actual value
of the field, not just the text that the user sees. For example, for the first choice in this field the value
would be "wild_bear", not "Black bear". "Black bear" is just the text the user sees - the label.
question: Favorite bear type
field:
- Pick one: users[i].favorite_bear.type
datatype: radio
choices:
- Black bear: wild_bear
- Teddy bear: cuddly_bear
- Jaime: maybe_not_a_bear
Special HTML
This HTML lets ALKiln set the correct variables and avoid other problems.
Most authors add this extra HTML to the post
section of their default screen parts
block. They do that by including the AssemblyLine's al_package.yml file or by adding this code themselves. For example:
default screen parts:
# HTML for interviews using ALKiln tests
# If your interview includes AssemblyLine's al_package.yml you can skip this
post: |
<div id="alkiln_proxy_var_values"
data-generic_object="${ encode_name(str( x.instanceName if defined('x') else '' )) }"
% for letter in [ 'i', 'j', 'k', 'l', 'm', 'n' ]:
data-index_var_${ letter }="${ encode_name(str( value( letter ) if defined( letter ) else '' )) }"
% endfor
aria-hidden="true" style="display: none;"></div>
<div id="trigger" data-variable="${ encode_name(str( user_info().variable )) }" aria-hidden="true" style="display: none;"></div>
This code will be invisible on the page. No one will see it. If you already have something in your post:
default screen part, just copy the code and paste it in after whatever you have there. Putting it at the end can avoid messing up other HTML.
Do I need this?
You probably need the "alkiln_proxy_var_values" HTML element if you use generic objects (x
) or index variables (i
, j
, k
, etc) in your interview.
You probably need the "trigger" HTML element if you set restrict input variables
to True
in your docassemble server configuration file.
Why x
and i
?
Without the help of the HTML, ALKiln has no way of knowing what the actual names are of variables that use generic objects (x
) or index variables (i
, j
, k
, etc).
Why "trigger"?
If a docassemble server's config value of restrict input variables
is set to True, ALKiln needs the help of the HTML to know what variable a signature Step is setting.
That said, even the "trigger" HTML may not help if the question is triggered by mandatory: True
.
Text
Text-type fields are fields where the user needs to type text. The default field type is a text field. Other text fields include those with:
datatype: date
datatype: time
datatype: currency
datatype: password
datatype: email
datatype: phone
datatype: text
datatype: integer
datatype: number
input type: area
fields are textarea fields and also accept text for their value
You can see more about datatype
s and input type
s in the docassemble docs.
Example text fields:
question: Name of your favorite bear
field:
- Bear name: users[i].favorite_bear.name.first
- Why is this one your favorite?: users[i].favorite_bear.why
input type: area
Example linear Steps:
And I set the variable "users[0].favorite_bear.name.first" to "Ultraman"
And I set the variable "users[0].favorite_bear.why" to "Why not?\nIt seems ok to like Ultraman."
Example Story Table Step:
| var | value |
| users[0].favorite_bear.name.first | Ultraman |
| users[0].favorite_bear.why | Why not?\nIt seems ok to like Ultraman. |
If an answer needs to have multiple lines, you can still only use one line in your test code. Instead use \n
where a new line would be. See examples.
Dates
Dates are a specific kind of text value. ALKiln expects the format ##/##/####. For example, 06/26/2015. For Assembly Line custom three-parts date fields, ALKiln will assume the date format is dd/mm/yyyy.
There is also a special ALKiln value named today
.
Today
One special ALKiln value you can include is today
. That will insert the date on which the test is being run. You can also subtract from, or add days to, today
.
Example linear Steps:
And I set the value "signature_date" to "today"
And I set the value "court_date" to "today + 20"
And I set the value "minors_birth_date" to "today - 3650"
Example Story Table Step:
| signature_date | today |
| court_date | today + 20 |
| minors_birth_date | today - 3650 |
The last example makes sure that the date is 10 years in the past, ensuring that a minor always stays a minor for that test.
Be careful with today
math. If you are testing a "boundary" date, you want to pick a date that is well within a boundary to make the test more robust. For example, if you want to test a date that was 10 or fewer days ago, use today - 9
instead of today - 10
.
At some point your tests might run over night and hit midnight right in the middle of the test - after ALKlin fills in the answer, but before the interview finishes. In that situation, if ALKiln answered the question with today - 10
the clock would tip into the next day by the end of the interview - 11 days from the filled-in date. Your test would fail incorrectly.
The fact is, your users also might have that exact experience, so keep that in mind as you design your interview. For example, instead of just telling a user that they are or are not eligible, you can tell them the date of the deadline so they too will know about the boundary.
yesno
There are a few different yesno
fields - yesno
buttons or yesno
fields like yesno
checkboxes and yesnoradio
s.
Example yesno checkbox field:
question: Bears
field:
- Do you have any bears?: has_bear
datatype: yesno
Example of setting the yesno checkbox to "yes" in a linear Step:
When I set the variable "has_bear" to "True"
Example of setting the yesno checkbox to "yes" in a Story Table Step:
| var | value |
| has_bear | True |
noyes
Avoid noyes
type fields. For one thing, using noyes
will make you edit the code you get from the ALKiln Story Table Step generator more. For another, we've found that humans tend to find those confusing too.
If you do need to use a noyes
field, you will have to think differently about the value here. Set the variable to True
if you want the box to be checked. Set it to False
if you want the box to be unchecked.
Example of making sure a noyes
checkbox field gets checked in a linear Step:
And I set the variable "likes_bears" to "checked"
or, in a Story Table:
| var | value |
| likes_bears | checked |
yesnomaybe
There are a couple different yesnomaybe
fields - yesnomaybe
buttons and datatype: yesnomaybe
fields.
Example yesnomaybe field:
question: Bears
field:
- Do you have any bears?: has_bear
datatype: yesnomaybe
Example of setting the yesnomaybe radio buttons to "maybe" in a linear Step:
When I set the variable "has_bear" to "None"
Example of setting the yesnomaybe radio buttons to "maybe" in a Story Table Step:
| var | value |
| has_bear | None |
radio
Example radio button field:
question: Favorite bear type
field:
- Pick one: users[i].favorite_bear.type
datatype: radio
choices:
- Black bear: wild_bear
- Teddy bear: cuddly_bear
- Jaime: maybe_not_a_bear
Always use options that have both a label and a value.
Example linear Step:
When I set the variable "users[0].favorite_bear.type" to "cuddly_bear"
Example Story Table Step:
| var | value |
| users[0].favorite_bear.type | cuddly_bear |
checkboxes
Checkboxes are one of the most challenging variables to set. This might lead to a "variable not found" error or an ALKiln Story Table Step might just skip the field causing problems later on.
- The only value a checkbox can have is
True
orFalse
. - In ALKiln, the variable name of the checkbox is usually the same as its docassemble variable name. For example,
good_fruits['Apples']
. - The names of
object_checkboxes
variable names in ALKiln are similar to how docassemble uses them, but they have extra quotes inside the first square brackets. For example:best_friends['all_friends[0]']
An example of a checkbox field:
fields:
- Out of all your friends, choose your best friends: best_friends
datatype: checkboxes
choices:
- Emmet: emmet
- Maria: maria
- Sam: sam
Always use options that have both a label and a value.
The variable name of the checkbox for ALKiln is usually the same as its docassemble variable name. For example, good_fruits['Apples']
. The only value a checkbox can have is True
or False
.
And I set the variable "best_friends" to "Emmet"
And I set the variable "best_friends['Emmet']" to "True"
or, in a Story Table:
| var | value |
| best_friends['Emmet'] | True |
One exception is field created with an object
datatype, like object_checkboxes
. Their ALKiln names are similar to their docassemble names, but they need extra quotes. For example, look at this field:
fields:
- Out of all your friends, choose your best friends: best_friends
datatype: object_checkboxes
choices: all_friends
And I set the variable "best_friends" to "all_friends"
And I set the variable "best_friends" to "all_friends[0]"
And I set the variable "best_friends[all_friends[0]]" to "True"
And I set the variable "best_friends['all_friends[0]']" to "True"
or, in a Story Table:
| var | value |
| best_friends['all_friends[0]'] | True |
Especially notice the quotes just inside the first square brackets.
To choose "none of the above" use the word 'None'
in your variable name:
And I set the variable "best_friends['None']" to "True"
or, in a Story Table:
| var | value |
| best_friends['None'] | True |
file
You can upload one or more files by setting a variable to a comma-separated list of file names to upload. You must store files that you plan to upload in your "Sources" folder along with your tests.
The format for a linear Steps:
And I set the variable "<variable name>" to "<comma separated list of a file or files>"
The format for a Story Table Step:
| var | value |
| <variable name> | <comma separated list of a file or files> |
Example of a linear Step to upload 1 file:
And I upload "refutable_evidence.jpg" to "evidence_files"
Example of a linear Step to upload multiple files:
And I upload "irrefutable_evidence.jpg, refutable_evidence.pdf" to "evidence_files"
Example of a Story Table Step to upload 1 file:
| var | value |
| evidence_files | refutable_evidence.jpg |
Example of a Story Table Step to upload multiple files:
| var | value |
| evidence_files | irrefutable_evidence.jpg, refutable_evidence.pdf |
To upload more than one file you must separate their names with a comma.
signature
Signatures look a bit funny in ALKiln at the moment, but they work fine.
Example of a docassemble signature page:
question: Sign your name
signature: user.signature
under: ${ user }
Example of linear Steps:
And I sign
# or
And I sign with the name "<signatory's name>"
The first version will show a single dot as the signature. The second version will show the name and also have a dot.
In a Story Table, a signature variable has nothing in its value
column. Example of a Story Table Step:
| var | value |
| user.signature | |
All Story Table signatures are a single dot.
.there_is_another
In a Story Table, setting the value of .there_is_another
for a docassemble loop is more complicated than you might expect. At first glance, to fill in the answers for two children some authors try to write the following:
| var | value |
| kids.there_are_any | True |
| kids.there_is_another | True |
Instead of using .there_are_any
and .there_is_another
, always use .target_number
in their place to set the number of items that you want to include in the test.
For 3 children:
| var | value |
| kids.target_number | 3 |
For no children:
| var | value |
| kids.target_number | 0 |
Observational Steps
You can use some Steps to look at what is on the page or compare data. You can use these to make sure the interview has done what you want it to.
I check the page for accessibility issues
Use the "accessibility" Step to check the current page for problems with accessibility.
Example:
Then I check the page for accessibility issues
If there are problems with the accessibility of the page, ALKiln will continue to the end of the test, but it will save a separate JSON accessibility report file for a page that fails this check. For example, if text is too small or if contrast between the text and its background is too low. If ALKiln found problems, the test will fail at the end.
ALKiln uses aXe-core to check for problems. You can check the documentation there to understand the failure data better. A human can do a much more thorough job of improving accessibility, but this can be a start.
This is different than the "all accessible" Step, which checks all the subsequent pages for accessibility issues.
I expect the baseline PDF "<baseline file>" and the new PDF "<new file>" to be the same
You can compare an example PDF (sometimes called a baseline) to the PDF your interview created to make sure they are the same. You must save the baseline PDF in your "Sources" folder along with your tests. Then use the "download" Step to download the file your interview created. You can then compare the two PDFs.
Then I expect the baseline PDF "baseline.pdf" and the new PDF "download.pdf" to be the same
First ALKiln will compare all of the text in the PDFs. Then ALKiln will compare the fillable fields in the PDFs. If anything is different, ALKiln will print the difference in the report. You can then use the text in the report to search through the PDFs and find the difference there.
I SHOULD see the phrase "<page text>"
Use the "phrase" Step to check that specific text is visible on the page. You can also make sure that text that should NOT be visible is indeed absent.
Getting some text characters right can be tricky with interview pages. If you get told ALKiln cannot find a phrase, read about possible reasons for missing phrases in the troubleshooting documentation.
The Step format for variation 1 - the phrase does exist on the page:
Then I SHOULD see the phrase "<page text>"
The Step format for variation 1 - the phrase is absent from the page:
Then I should NOT see the phrase "<page text>"
The blank to fill in here is <page text>
- the phrase to test.
Examples for variation 1 - the phrase does exist on the page:
Then I SHOULD see the phrase "Yay! This page's text is great!"
Then I should NOT see the phrase "Boo! This page's text is bad!"
The <page text>
must be inside double quotation marks and must never itself contain double quotation marks inside it. Since docassemble interview text usually has special quotation marks, that is usually easy. See the troubleshooting section on missing phrases that talks about special characters.
Unlike many other Steps, checking phrases is language specific. That is, if you want to test translations as well, you will have to use an "Examples" table in your test.
the text in the JSON variable "<variable name>" should be
Use the "compare JSON" Step to check that a variable on the page has a specific text value. This is a multi-line step. It is a little limited at the moment.
The Step format:
Then the text in the JSON variable "<variable name>" should be
"""
<value>
"""
The first blank to fill in for this Step is <variable name>
which is the name of the variable that is in the JSON variables and values data of the page. The second blank to fill in is <value>
, which is the value you want the variable to have. The value must be a string (text).
This step is unable to check values of nested objects. For example, it can test the value of a variable like user_affidavit
, but not an attribute like user.affidavit
.
Example for one line of text:
Then the text in the JSON variable "nickname" should be
"""
Ursus
"""
Example for text with multiple lines:
Then the text in the JSON variable "user_affidavit" should be
"""
Three quotes then some affidavit text.
The text can be multi-line.
Then close with three quotes.
"""
This Step will save a copy of all of the page's JSON variables to a file that starts with 'json_for' followed by the current question's id. The JSON variables are the same variables that you would see in the docassemble sources tab.
Avoid JSON Steps when you have added sensitive information to your interview.
Variable values only show up in the JSON on the next page - the one after the page where you fill in the answers.
I take a picture
The "picture" Step will not only take a picture of the page, it will also download an HTML file of the page. The associated HTML file will have the same name as the picture, but will end with .html
. ALKiln will save those files in the Scenario folder of the test results.
You can open the HTML file in your browser to interact with the page and inspect the page's HTML further. You should expect this page to look and act different than the original page - ALKiln is unable to get the full code that controls the behavior and styles of the original page. However, in the HTML, you can look at what options were in a drop down, examine some accessibility errors, or edit the HTML to expand some collapsed content. In fact, some pages are too long for the picture to show all the content. In the HTML, you can see that content.
Example:
Then I take a picture
Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables. It even avoids taking pictures when the test has an error on that page and during the sign-in Step. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.
I get the page's JSON variables and values
Use the "get JSON" Step to add a page's JSON variables to the final test report.
Example:
And I get the page's JSON variables and values
The JSON variables are the same variables that you would see in the docassemble sources tab. It is a bit messy, but you do get to see all the variables.
Avoid JSON Steps when you have added sensitive information to your interview.
Variable values only show up in the JSON on the next page - the one after the page where you fill in the answers.
I will be told an answer is invalid
Use the "invalid answer" Step to check that the user got feedback that one or more of their answers were invalid.
Example:
Then I will be told an answer is invalid
the question id should be "<id>"
Use the "question id" Step to make sure the test has gotten to the right page.
The Step format:
Then the question id should be "<question id>"
The blank to fill here is <question id>
- the id of the question
block for this page. To get the right <question id>
, copy the id
value from that block and paste it into your test.
Example:
Then the question id should be "some YAML block id!"
This Step can help humans keep track of what page the tests are on. It will also show up in the test reports and can help you see where things went wrong.
Make constrained random tests
Only for GitHub tests at the moment.
You can make template .feature
files that each generate multiple tests with different combinations of answers. On GitHub, generated tests will be in your GitHub artifacts folder. You can download them and copy them into your test suite if you want.
When to use random tests
Use random tests to:
- Explore paths you didn't think of
- Generate test ideas
- Build up a strong set of test suite files by downloading and saving generated tests
Avoid random tests for:
- Safety concerns like impounding addresses
- Important edge-cases5
- Bugs you already know about
- Other tests you need to make sure you run regularly
Hand-written tests use your deep knowledge of the code and your human intelligence to reliably and consistently test known critical paths and edge cases. They are your strongest tool. Still, random tests can add be a good supplement.
Example of the template file named alkiln_random_no_children.feature
for users with no children:
Feature: Random no children tests
@no_children
Scenario: I have no children
Given ALKiln will make 2 constrained random answers tests
And I start the interview at "package.yml"
And I wait 1 second
Then ALKiln will reach any of the question ids "encouragement 1" or "wrong State kickout" with:
| var | possible values |
| number_of_children | 0 |
| lives_in_MA | True;; False |
| is_married | True;; False |
| spouse_hair_color | red;; brown;; blue;; other |
| spouse_other_hair_color | magenta |
And I take a picture
Then ALKiln will reach any of the question ids "no children end" with:
| var | possible values |
| impound_address | True;; False |
| wants_a_translator | True;; False |
| translation_language | Spanish;; Portuguese;; Mandarin |
And I take a picture
Example of the newly generated file, alkiln_generated_1_1744635556587.feature
:
@alkiln @generated @randomized @constraints
Feature: Random no children tests
@no_children @scenario1_of_2
Scenario: I have no children (1 of 2)
And I start the interview at "package.yml"
And I wait 1 second
And I get to any of the question ids ["encouragement 1", "wrong State kickout"] with this data:
| var | value |
| number_of_children | 0 |
| lives_in_MA | False |
| is_married | False |
| spouse_hair_color | red |
| spouse_other_hair_color | magenta |
And I take a picture
And I get to any of the question ids ["no children end"] with:
| var | value |
| impound_address | True |
| wants_a_translator | False |
| translation_language | Portuguese |
And I take a picture
@no_children @scenario2_of_2
Scenario: I have no children (2 of 2)
And I start the interview at "package.yml"
And I wait 1 second
And I get to any of the question ids ["encouragement 1", "wrong State kickout"] with this data:
| var | value |
| number_of_children | 0 |
| lives_in_MA | True |
| is_married | False |
| spouse_hair_color | other |
| spouse_other_hair_color | magenta |
And I take a picture
And I get to any of the question ids ["no children end"] with:
| var | value |
| impound_address | True |
| wants_a_translator | True |
| translation_language | Mandarin |
And I take a picture
Key rules:
- Name your template files starting with
alkiln_random
- Put only ONE scenario in each template file
- Tell ALKiln how many tests to create
- List at least one answer for every required field for every page ALKiln might visit
During the test run, ALKiln will generate the tests and then run them.
Failing: A random test fails if ALKiln gets stuck or runs into a System error. For example, the interview might get an "Index out of range" error.
Passing: A random test that passes can still be wrong. The report will have more information. For example, you can make sure:
- ALKiln visited the right pages for the answers it gave
- ALKiln reached the right page ids
- The report doesn't have any unexpected warnings
ALKiln will make <number>
constrained random answers tests
<number>
constrained random answers testsUse a "test count" Step in a template file to tell ALKiln how many tests to make. Example:
Given ALKiln will make 10 random constrained tests
ALKiln will avoid making duplicate tests6 even when ALKiln generates the tests from separate template files. If necessary, ALKiln will make fewer tests and add a warning Scenario to the generated file. You can see it in the report.
ALKiln will reach "<question id 1>
" or "<question id n>
" with:
<question id 1>
" or "<question id n>
" with:Use the "constraints" Steps in a template file to tell ALKiln:
- What answers it must choose from to build the random tests
- Where to finish this constraints Step
ALKiln will turn every constraints Step into one or more Story Tables.
Example of a constraints Step:
Then ALKiln will reach "encouragement 1" or "wrong State kickout" with:
| var | possible values |
| number_of_children | 0 |
| lives_in_MA | True;; False |
| is_married | True;; False |
| spouse_hair_color | red;; brown;; blue;; other |
| spouse_other_hair_color | magenta |
Example of a Story Table ALKiln might generate:
And I get to any of the question ids ["encouragement 1", "wrong State kickout"] with this data:
| var | value |
| number_of_children | 0 |
| lives_in_MA | False |
| is_married | False |
| spouse_hair_color | brown |
| spouse_other_hair_color | magenta |
There are two parts to the constraints Step—the Step text and the fields table.
-
The Step text has one type of value to fill in—1 or more page ids. Each page id must be in double quotes (
"
). This is like a Story Table—the page ids tell ALKiln when to finish with this Step. -
Every row of the fields table has 2 columns. The first row must be the header row:
| var | possible values |
For every other row:
- The 1st column must name the variable to set, just like in a Story Table.
- The 2nd column must have 1 or more values that ALKiln can choose from. If there is more than 1 value, you must use 2 semicolons (
;;
) to separate the values.
You can have as many of these “constraints” Steps in a template Scenario as you want. You can put any other steps you want in between those “constraints” Steps. You can put any kind of Step you want after your last “constraints” Step.
You must define all variables that the interview might require ALKiln to answer
Note that the example above gives options for spouse_hair_color
even though the user can say they don't have a spouse. If the user is married spouse_hair_color
is a required field and ALKiln will need to know the possible answers for that field.
Don't worry about including rows that might not get used. ALKiln is just fine with skipping extraneous table rows.
If a field is optional and you don't want ALKiln to give an answer for it, leave out that row.
Tip: You can use strict limitations to focus on important paths
You can force the tests to focus on a specific path by only putting one option in a possible values
column. You can do this to as many rows as you want.
You can see the example above only gives one option for number_of_children
—0
. The author of this test wants to make sure ALKiln makes a good number of random tests that are just about users who have no children. The author might make another template file to generate random tests for users that have 2 children.
;;
) at the end of the choices. It will make a choice that is an empty string. If you deliberately want to include that option, include it in the middle of your choices instead.| favorite_bear | teddy;; grizzly;; |
| favorite_bear | teddy;; ;; grizzly |
Warnings for templates
ALKiln will throw an error if the file or Scenario is invalid. There are some typos or misbehaviors that ALKiln will only warn you about, though. Some examples:
- Your Scenario has no digit value for the test count. ALKlin will make 1 test.
- Your Scenario asks for more than the maximum number of tests. You can set this number with the
ALKILN_MAX_RANDOM_TESTS_PER_SCENARIO
environment variable. - ALKiln tries to make the number of tests you asked for, but it is unable to keep them all unique so it makes fewer tests.
ALKiln will put a "warning Scenario" at the top of the file it generates. You can also see it in your test report. Example warning:
@alkiln_extra_intro @alkiln_generator_warning
Scenario: ALKiln begs the author's forgiveness. The next tests might be flawed.
Then ALKiln warns the author about a generator flub:
"""
――――― Template file warnings ―――――
🔎 ALK0239 generating files WARNING: A generator Scenario is missing the number of tests to make. ALKiln will now make 1 constrained random answers test. The current Step:
Given ALKiln will make constrained random answers tests
―――――
"""
Convert an interview into a Story Table
You can use the Story Table converter to create a Scenario draft based on the data from an interview you have gone through. Depending on your interview's code you might need to edit the new table for it to work properly, but it can give you a good start.
Follow these instructions to use the converter:
- Make sure your server config is set up to show debug info.
- Run your interview manually until you reach the page you want the story table to get to.
- Open the "source" display of the interview. Currently, that looks like angle brackets,
</>
, in the header of the page. - Note the
id
of the page. - Tap the "Show variables and values" button. It will open a new tab showing a big JSON object.
- Copy all the text on that page.
- Go to the Story Table converter.
- Paste the JSON into the text area there, as instructed.
- Use the other input fields to help finalize your Scenario, including the question
id
. - Download the file the converter created. It should end in
.feature
. - Upload that
.feature
file into your "Sources" folder in your Project.