Customer Guide (Specification Guide)



Please Note:

The following executable tables are designed with a header row of 2 columns. This is just design. You can create tables with e.g. 3x3 cells.

Writing my first executable specification

Writing an executable specification for a Calculator program

Consider your customer wants to create a calculator.

The business expert on Calculators explains to the developer the rules for calculation. The first specification is: The calculator should be able to add two numbers. To support the specification, the business expert also provides an example: 6 + 3 = 9.

In order to be executable in , the specification must be expressed in a specific form :

Decision TableCalculator
xysum?
639

The first line of the specification explains that we have particular RULES FOR our Calculator. The calculator receives two numbers and gives back the sum of those two numbers. Once the calculator is developed, we will be able to execute the specification to test the functionality.

Execute the specification

If you hit the Execute button on the top of the page, you will get the result of your test.

How it works?

Once you click on the Execution button,  calls the actual software containing the code for the Calculator. It sends the parameters x (equal to 6) and y (equal to 3) to the Calculator to obtain the resulting sum of those two numbers. The "?" symbol next to the word "sum" is a key-sign, that indicates to  that a test must be performed.

In that particular example,  will query the calculator application and compare the value returned by the system with the value specified by the business expert (in the "sum?" column). Since the result returned by the system is the same as the expected result, the cell is colored in green by . The business expert is now confident that the Calculator is able to add two numbers.

Adding another example

Once the Calculator is developed, the business expert may think about new examples that he would like to test. The business expert adds another row in the Decision Table table and executes the test (no coding is needed here).

Decision TableCalculator
xysum?
639
708

When there is an error,  colors the cell in red.
In the new example, it is clear that the expected value provided by the business expert is not correct. But the error could have been in the system.

The business expert should always be consulted before any changes are made in the executable specifications

Adding new specifications to our Calculator

The business expert wants to improve the Calculator: The Calculator should be able to perform product and quotient of two numbers.
The new rules can be written in a way similar to the "sum" specification. In case of expected errors, the keyword error may be used in order to test those special cases.

Decision TableCalculator
xysum?product?quotient?
628123
7070error

Rule Validation (Decision Table)

Definition

DEFINITION

The RuleForInterpreter is used to express concrete and measurable business rules.
During the execution of the specification,  compares the values returned by the system under development against the expected values defined by the Business Expert.

  • The first row of the table indicates the set of rules to be tested by .
  • The next row is called the header row and serves to distinguish the given values and the expected values. Given values serve as input to the system, whereas expected values serve as comparison values against values that are actually returned by the system. When a column header ends with special characters ? or (), it denotes an expected value.
  • Finally, the remaining rows capture the examples. They are the executable test rows.

Specific Keywords for expected values

 offers a list of useful keywords to support the Business Expert.

Empty cells
When a test cell is left blank,  only shows the returned value
errorWhen you expect an error, specify it in the cell to test that particular behavior

Coloring

 will visually show the test result by coloring each testing cell:


Green

 When the expected value matches the returned value, the RuleForInterpreter colors the cell as "right" by coloring it green.

Red

If the values don't match, the RuleForInterpreter colors the cell as "wrong" in red.

YELLOW

If the system encounters an execution error, the cell is colored yellow and  provides information about the error.

Grey

If no expected value is specified in a test, the RuleForInterpreter colors the cell in gray.


Here is an example of cell coloring:

Input Table:

Decision TableCalculator
xysum?product()quotient?
1001000
639-5

Output Table:

Decision TableCalculcator
xysum?product()quotient?
100100

0


java.lang.ArithmeticException: / by zero
info.novatec.testit.livingdoc.fixture.calculator.Calculator.quotient(Calculator.java:63)
639Expected:  -5  Received:  182


Writing a Decision Table Specification

When do we use the RuleForInterpreter?

The RuleForInterpreter is used to express concrete and measurable business rules. The business rules are often related to all kind of calculations.

When a table of rule is executed,  compares the values returned by the system under development against the expected values defined by the Business Expert.

First row: Identification of the Decision Table

In a Decision Table table, the first row specifies the name of the interpreter and the name of the rules to be tested.
The business expert should be clear in the identification of the set of rules.

  • Example Division : A simple system performing divisions. In that particular case, we could define the set of rules as Division which leads to:

    Decision TableDivision
  • Example Mortgage : A Mortgage management system with the following specifications:
    "Whenever the down payment on the property is less than 25% of the sale price, the buyer has to pay a premium for insuring the mortgage. The premium fee amount is 2.5% of the financed amount (which is sale price - down payment)."
    For that particular calculation we could identify the set of rules as:

    Decision Tableinsurance premium fee calculation

Second row: the Header

The second row is called the header row and serves to distinguish between the given values and the expected values. Given values serve as inputs to the system, whereas expected values serve as comparison values against values that are actually returned by the system. When a column header ends with special characters ? or (), it denotes an expected value.

  • Example Division : The given values for that example are the dividend and the divisor. The expected value is the quotient.

    Decision TableDivision
    dividenddivisorquotient?
  • Example Mortgage : The given values for that example are the sale price and the down payment. The expected value is the premium fee.

    Decision Tableinsurance premium fee calculation
    sale pricedown paymentpremium fee?

Third and following rows: Examples

The remaining rows capture the examples. They are the executable test rows.

Final expression of the Division example

Decision TableDivision
dividenddivisorquotient?
6.02.03.0
723.5
183.06
1528

Final expression of the Mortgage example

Specification: "Whenever the down payment on the property is less than 25% of the sale price, the buyer has to pay a premium for insuring the mortgage. The premium fee amounts is 2.5% of the financed amount (which is sale price - down payment)."

Decision Tableinsurance premium fee calculation
sale pricedown paymentpremium fee?
$100,000$15,000$2,125.00
$100,000$30,000$0.00
$100,000$25,000$0.00

When there is no expected value specified in a cell,  provides the value calculated by the system under development. The business expert could use that functionality to facilitate the comprehension of a rule. In this example, the financed amount of mortgage can be evaluated as an intermediate result.

Decision Tableinsurance premium fee calculation
sale pricedown paymentpremium fee?financed amount?
$100,000$15,000$2,125.00
$100,000$30,000$0.00
$100,000$25,000$0.00

Execution of specifications

Based on those executable specifications, the developers are now ready to code the functionality and the fixture (the fixture is the link between the system under development and the executable specification). Once this is done, the specification can be executed by clicking on the Execute button on the top of the page.

During execution,  colors only the cells of the column "expected values".  will color the cell green if the returned value equals the expected value. If the returned value is not the same,  will color the cell red and will show the expected and the returned values. Finally, if an expected value is left blank by the business expert,  colors the cell in gray, to display that no test was performed, and shows the returned value.

The examples provided in this page are used to explain how to write the fixture for the RuleForInterpreter.

List Validation (List of, Set of, Superset of, Subset of)

Definition

DEFINITION

The collection interpreters are used to express any kind of groups, lists or sets of values.

When a collection interpreter is executed,  compares the list of values expected with the list of values returned by the system under development. The test result depends on the specific collection interpreter selected.

  • As for all other interpreters, the first row of the collection interpreters specifies the name of the interpreter and the name of the collection to be tested.
  • The next row is used to define the name of each attribute related to the members of the collection of data.
  • The following rows are used to specify the set of values to test.

Specific keywords

None

Coloring

 will visually indicate the test result by coloring the rows:

Green

 As expected, the row is returned by the system under development.

Red

A row is missing or in surplus in the list returned by the system under development.

YELLOW

If the system encounters an execution error, the cell is colored in yellow, and  provides information about the error.

Particular behavior for the list of interpreters:
Since the ListOfInterpreter expects to match exactly the expected list and the list returned by the SUD,  compares each cell of the table separately. If the expected value is different from the returned value,  colors the cell in red and provides the two values.

Particular behavior for the superset of interpreters:
Since the SupersetOfInterpreter accepts that the specification may contain rows that are not contained in the SUD,  will not color the rows, which are contained in the specification but not returned by the SUD.

Writing a List specification

When do we use the CollectionInterpreters

The CollectionInterpreters are used to express any kind of group, list or set of values.

When a CollectionInterpreters is executed,  compares the list of expected values with the list of values returned by the system under development. The test result depends on the specific CollectionInterpreters selected.

 offers four different CollectionInterpreters:

ListOfInterpreterThis interpreter expects to match the content and the order of the list returned by the system under development (SUD) with the expected list defined in the specification.

A is a subset of B
B is a superset of A

SetOfInterpreterThe SetOfInterpreter expects to match only the content of the list returned by the SUD with the expected list defined in the specification. The order of the rows is not important.
SubsetOfInterpreterThe SubsetOfInterpreter verifies that each row defined in the specification exists in the list returned by the SUD. In order to have a successful test, the set of elements listed in the specification should be a subset of the list returned by the SUD.
SupersetOfInterpreterThe SupersetOfInterpreter verifies that each row returned by the SUD exists in the list defined in the specification. In order to have a successful test, the set of elements listed in the specification should be a superset of the list returned by the SUD. This means that the specification may contain rows that are not contained in the SUD.

First row: Identification of the Collection Interpreter

As for all other interpreters, the first row of any CollectionInterpreter specifies the name of the interpreter and the name of the collection to be tested.

  • Example - Provinces of Canada: A simple example would be a system containing the Canadian provinces name and code. In that particular case, we could define the collection as Canada Province Codes :

    list ofCanada Province Codes
  • Example - Phone Book : A phone book application able to manage the first name, last name and the phone number of people. The ListOfInterpreter could be expressed as :

    list of

    phone book entries

Other rows: A set of elements

From the third row, we specify the set of values to test.

  • Example - Provinces of Canada :

    list ofCanada Province Codes
    NameCode
    ALBERTAAB
    BRITISH COLUMBIAAB
  • Example - Phone Book :

    list ofphone book entries
    FirstNameLastNamePhone Number
    FredFlintstone(123) 456-7890
    BarneyRubble(123) 321-7666
    GreatGazoo(123) 989-4455

Execution of specification

Once the developers have coded the functionality and the fixture(the fixture is the link between the system under development and the executable specification), the specification can be executed by clicking on the Execute button on the top of the page.

During execution,  will color the rows in green if the rows are in the list of elements returned by the SUD as expected in the specification.
If a row is missing or in surplus in the list returned by the SUD, then  will color the entire row in red and specify if the row is  missing  or in  surplus .

Notes

If  specifies that a row is missing , it means that the row is not in the list of elements provided by the system under development as it should be.
If  specifies that a row is in surplus , it means that the row is in the list of elements provided by the system under development, as it should not be.

Since the ListOfInterpreter expects to match exactly the list expected and the list returned by the SUD,  compare each cell of the table separately. If the expected value is different from the returned value,  colors the cell in red and provides the two values.

Since the SupersetOfInterpreter accepts that the specification may contain rows that are not contained in the SUD,  will not color the rows, which are contained in the specification but not returned by the SUD.


Executable specification for the example - Provinces of Canada

list ofCanada Province Codes
NameCode
ALBERTAAB
BRITISH COLUMBIABC
MANITOBAMB
NEW BRUNSWICKNB
NEWFOUNDLAND and LABRADORNL
NOVA SCOTIANS
NUNAVUTNU
ONTARIOON
PRINCE EDWARD ISLANDPE
QUEBECQC
SASKATCHEWANSK
YUKONYT
OTHER PROVINCEOP

Executable specification for the example - Phone Book

Creating the collection of data
We will use the DoWithInterpreter to create our personal phone book.

Workflowphone book
insertFred
Flintstonewith number(123) 456-7890
insertBarney
Rubblewith number(123) 321-7666
insertGreat
Gazoowith number(123) 989-4455

ListOfInterpreter : The requirement list should be the same as the SUD list.

list ofPhone book entries
FirstNameLastName
FredFlintstone
BettyRubble
GreatGazoo
WilmaFlintstone

SetOfInterpreter : The requirement set should be the same as the SUD set.

set ofPhone book entries
FirstNameLastName
FredFlintstone
BettyRubble
GreatGazoo
WilmaFlintstone

SubsetOfInterpreter : The requirement set should be a subset of the SUD set.

subset ofPhone book entries
FirstNameLastName
FredFlintstone
BettyRubble
GreatGazoo
WilmaFlintstone

SupersetOfInterpreter : The requirement set should be a superset of the SUD set.

superset ofPhone book entries
FirstNameLastName
FredFlintstone
BettyRubble
GreatGazoo
WilmaFlintstone

The examples provided in this page are used to explain how to write the fixture for the CollectionInterpreters.

Workflow validation

Definition

DEFINITION

The DoWithInterpreter is used to express interactions with the system under development that must be performed in a particular order. This form of specification provides information about the business flow.

When a sequence of action is executed,  confirms that each action has successfully been performed.

  • As for all other interpreters, the first row of the DoWithInterpreter specifies the name of the interpreter and the name of the sequence of actions to be tested. What makes the DoWithInterpreter particular is that it only has to be defined once for all the sequences of actions expressed in a page. Obviously, the DoWithInterpreter must be defined before any sequence of actions.
  • The following rows are used to express specific actions.
  • The form of each row of a DoWithInterpreter shall respect the following rules:
    • a row shall begin with a part of the action description,
    • each parameter shall be isolated in a cell,
    • each parameter shall be separated by parts of the action description.
  • An action description can be left blank in order to separate two parameters.
  • The DoWithInterpreter provides a minimum of keyword used to define a specific action.
  • The DoWithInterpreter may also be expressed in Bullet List form or Number List form.

Specific Keywords

 offers a list of useful keywords to support the Business Expert. Those keywords are placed at the beginning of an action row.

AcceptConfirm that the action has been executed by the system under development.
CheckVerify the specified expected value with the value returned by the system under development
RejectThe action should not be performed by the system under development (expected errors).
DisplayPrint the value returned by the system under development.

coloring

 will visually show the test result by coloring each testing cell:

Green

When the action has been executed successfully,  colors the cell(s) in green.

Red

If the action execution has failed,  colors the cell(s) in red.

YELLOW

If the system encounters an execution error, the cell is colored yellow and  provides information about the error.

Grey

When the action has been executed successfully,  will display the returned value in gray.

Standard form (without keyword)Only the Action description will be colored.
AcceptOnly the cell containing the keyword Accept will be colored.
Check
The cell containing the expected value will be colored. In the case of a failure,  will show the expected and the returned values.
RejectOnly the cell containing the keyword Reject will be colored.
DisplayA new cell at the end of the row will be colored containing the returned value.

Writing a Workflow specification

When do we use the DoWithInterpreter

The DoWithInterpreter is used to express interactions with the system under development that must be performed in a particular order. This form of specification provides information about the business flow.

When a sequence of action is executed,  confirms that each action has successfully been performed.

First row: Identification of the Workflow

As for all other interpreters, the first row of the DoWithInterpreter specifies the name of the interpreter and the name of the sequence of actions to be tested. What makes the DoWithInterpreter particular is that it only has to be defined once for all the sequences of actions expressed in a page. Obviously, the DoWithInterpreter must be defined before any sequence of actions.

Example of a bank Account management system

Example context :  The system under development must be able to manage two different types of bank account. The system should allow to execute the usual transactions within an account.

So the first line could be expressed as

Workflowbank account management

or more simply as

Workflowbank

Following rows: Actions & Verifications

The second and the following rows are used to express specific actions or verifications. The business expert should express the action in human readable format rather than using computer like language.

The form of each row of a DoWithInterpreter must follow these rules:

  • a row begins with a part of the action description.
  • each parameter is isolated in a cell
  • each parameter is separated by parts of the action description.
  • an action description can be blank

More visually:

Action description part 1parameter 1Action description part 2parameter 2Action description part 3Parameter 3...

Example: In our bank example, the first action would be to create a bank account.

If we want to take the account number as a parameter, we would have:

open checking account12345-67890

But, if we want to consider the type of account, the account number, the owner first name and last name as parameters, we would have:

opencheckingaccount12345-67890under the name ofSpongebob
Squarepants

Example: Verify the balance of an account

To verify the success of a particular action, we write the keyword "Check" prior to the first part of the action description.
In our example, there would be two parameters: the account number and the amount of balance.

checkthat balance of account12345-67890is$0.00

Final expression of our example

Workflowbank

Our first business rule says that a new account should have a balance of 0.00 dollars.

opencheckingaccount12345-67890under the name ofSpongebob
Squarepants
checkthat balance of account12345-67890is$0.00

Our next rule says that the bank should not take any fees when we deposit money in our account.

deposit$100.00in account12345-67890
checkthat balance of account12345-67890is$100.00

The following rule says that a customer should be able to withdraw funds, if the balance of his account is sufficient.

withdraw$50.00from account12345-67890
checkthat balance of account12345-67890is$50.00
rejectwithdraw$75.00from account12345-67890
checkthat balance of account12345-67890is$50.00
acceptwithdraw$25.00from account12345-67890

Execution of specification

Based on those executable specifications, the developers are now ready to code the functionality and the fixture (the fixture is the link between the system under development and the executable specification). Once this is done, the specification can be executed by clicking on the Execute button on the top of the page.

During execution,  will color cells in green if the execution has passed and color it in red if the execution has failed. The colored cells depends on the type of action described:

  • in a standard action row (without a keyword at the beginning), only the Action description will be colored,
  • if the action begins with the keyword Check, the value tested will be colored and in the case of a failure,  will show the expected and the returned values,
  • if the action begins with the keywords Reject or Accept, only the cell containing the keyword will be colored.
  • if the action begins with the keywords Display, only a new cell at the end of the row containing the returned value will be colored.

The examples provided in this page are used to explain how to write the fixture for the DoWithInterpreter.


Execution of specification

Based on those executable specifications, the developers are now ready to code the functionality and the fixture (the fixture is the link between the system under development and the executable specification). Once this is done, the specification can be executed by clicking on the Execute button on the top of the page.

During execution,  will color rows or words inside a row in green if the execution has passed and color it in red if the execution has failed.

The examples provided in this page are used to explain how to write the fixture for the ScenarioInterpreter.

6. Context definition (Setup)

Definition

DEFINITION

The SetUpInterpreter is used to simplify the creation of a particular state for the system under development. Once the state is created, we can focus on the business process to test.

When a setup table is executed,  enters data in the system under development to create the desired state.

  • The first row of the table indicates the name of the interpreter and the name of the desired state.
  • The second row is called the header row and serves to identify the data to be inserted in the system under development.
  • Finally, the remaining rows capture the data to be inserted.

Coloring

 will visually show the test result by coloring each testing cell:

Green

When the insertion has been executed successfully,  adds a green cell at the end of the data row with the word entered.

Red

If the insertion has failed because the values specified does not respect a business rule,  adds a red cell at the end of the row with the word not entered.

YELLOW

If the insertion has failed because a specified value generates an error,  colors the cell of the data in error yellow and provides information about the error.
If the system encounters an error not related to a specific data,  adds a yellow cell at the end of the data row and provides information about the error.

Writing a Setup specification

When do we use the SetUpInterpreter

The SetUpInterpreter is used to simplify the creation of a particular state for the system under development. Once the state is created, we can focus on the business process to test.

When a setup table is executed,  enters data in the system under development to create the desired state.

First row: Identification of the Set Up

As for all other interpreters, the first row of the setup table specifies the name of the interpreter and the name of the desired state.

Example of a bank Account management system

Example context : A customer should be allowed to transfer money from his or her accounts.

In order to create examples that would test this particular business rule, we need to create a set of customer. The first line of our setup table could be expressed as

setupa group of customers

Second row: the Header

The second row is called the header row and serves to identify the data to be inserted in the system under development.

Example: In our bank example, the header row could take the following form.

setupa group of customers
typenumberfirst namelast namebalance

Following rows: Data

The remaining rows capture the data, which is meant to be inserted. They are the executable rows.

Final expression of our example

setupa group of customers
typenumberfirst namelast namebalance
checking11111-11111    FredFlintstone$250.00
savings22222-22222    FredFlintstone$10 000.00
savings44444-44444    WilmaFlintstone$10 000.00
checking44444-44444    BarneyRubble$999.54
checking55555-55555    GreatGazooabc

From that point, we can now used those accounts to specify examples that define a business rule.

Workflowbank

Transfer from a savings account to a checking account of the same customer

acceptTransfer$1 000.00from account22222-22222    to account11111-11111    
checkthat balance of account11111-11111    is$1 250.00
checkthat balance of account22222-22222    is$9 000.00

Execution of specification

Based on those executable specifications, the developers are now ready to code the functionality and the fixture (the fixture is the link between the system under development and the executable specification). Once this is done, the specification can be executed by clicking on the Execute button on the top of the page.

During execution,  confirms that the value has been entered by adding a green cell at the end of each data rows containing the word entered.

If the data could not have been entered, because it does not respect a known business rule, the cell will be yellow and will specify the stacktrace. Finally, if the data could not be entered, because of an unknown error,  will add a yellow cell at the end of the row containing information about the exception. Note that if the error is specific to a data provided,  will color this specific cell yellow and will include the information about the exception.

The examples provided in this page are used to explain how to write the fixture for the SetupInterpreter.

Advanced

Workflow - Table, Bullet List or Number List example

 supports the expression of rules in table format, bullet list format or number list format. The following examples are equivalent and will yield the same results when executed.

It is important to note that the table and bullet list examples are not executable, since the import info.novatec.testit.livingdoc.samples.fixture.bank and Workflow bank rules cannot be specified twice in the same page. Even though it is possible to change between the two methods in the same page, you must keep in mind that the tests are executed in the same system under development. So it is not necessary to re-specify the same rule in the page.

Table

import
info.novatec.testit.livingdoc.samples.application.bank
Workflowbank
opencheckingaccount12345-67890under the name ofSpongebob
Squarepants
checkthat balance of account12345-67890is$0.00
deposit$100.00in account12345-67890
checkthat balance of account12345-67890is$100.00

Bullet List

  • import
  • info.novatec.testit.livingdoc.samples.application.bank


  • Workflow bank
  • open checking account 12345-67890 under the name of Spongebob _Squarepants
  • deposit $100.00 in account 12345-67890
  • check that balance of account 12345-67890 is $100.00

Number List

  • import
  • info.novatec.testit.livingdoc.samples.application.bank


  1. Workflow bank
  2. open checking account 12345-67890 under the name of Spongebob _Squarepants
  3. deposit $100.00 in account 12345-67890
  4. check that balance of account 12345-67890 is $100.00

Object-Graph Navigation Language (OGNL)

 has an extension to supports OGNL expressions. Here are some examples, using the Calculator system:

< , <= , > , >= , and , or
 supports these simple mathematical expressions. Use the question mark to specify the position of the value returned by the system. For example, if the expected value should be greater than 15, you would write "? > 15" in the cell.
+ , - , / , *
 performs simple operations as a resulting expression. To specify you want to evaluate those operations prior to the comparison, insert an equal sign at the beginning of the operation (ex: =4+1-3).
Decision Tablecalculator
xysum?product?quotient?
628123
7070error
405090? > 100
4-5? <= 0-20? <= 0
33? < 1 or ? > 4? >= 9 and ? < 12"1"
42=4+2-0=2*4=4/2

To use this extension, you need to specify the fixture factory as info.novatec.testit.livingdoc.extensions.ognl.OgnlSupport and add the livingdoc-extensions-ognl-x.x.jar and ognl-x.x.x.jar to the classpath of the SUT.

Information Tag

Information tags

When a document is executable,  will try to interpret all the supported forms of the document body.
This could lead to some issues, if we need to insert information as Tables or Bullet lists, for example.

To avoid undesired interpretation of forms, we can use the Information tags.
This tag will force  to skip all forms between the Begin Info and End Info tags.

Here is an example:

Begin Info
This tableshouldn't beinterpreted
  • This bullet list
  • shouldn't be
  • interpreted either
End Info

We won't see any coloration of the above table or bullet list after execution of this document.

 will restart the interpretation of the document from from this point.

Decision TableCalculator
xysum?
123

Useful Information

If the End Info tag is omitted all the body content of the document after the Begin Info tag, will be skipped.

Commented forms

Comment tag

If necessary, we can decide that a test shouldn't be executed. Instead of deleting the test from the document, we can comment it. This will force  to skip the test.

The test should be preceded by the tag Comment.

Here are some examples:

Test using the bullet list form.

  • Comment
  • My bulletlist test to skip

Test using the Table form.

Comment
My tableto skip

We won't see any coloration of the above table or bullet list after execution of this document.

 will restart the interpretation of the document from from this point.

Decision TableCalculator
xysum?
123