Introduction

Software testing is one of the most efficient ways to ensure delivery of high quality software at a rapid pace. It is an imperative step when onboarding new developers and increases overall confidence among all stakeholders.

Why we write tests

If the intro wasn't convincing enough:

• Manual testing and verification quickly gets ridiculously inefficient.
• A well-designed set of automatic tests with good coverage gives developers the confidence to add, modify, or remove code without fear.
• Testing allows you to maintain long term velocity in a project. It is easy to achieve good initial velocity by cutting corners, but those corners have to be taken care of eventually, and that is hard to do without tests that verify the existing functionality. If you keep adding things without going back and adapting existing code, the project will become a tangled mess and grind to a halt eventually.
• Well-written tests double as a form of always-up-to-date step-by-step documentation.
• Tests reduce cognitive bias limitations (i.e. "this worked five minutes ago, I don't need to test it again.", "feature A worked, therefore feature B must also work", or "there are never bugs in Amanda's code, we don't need to test it.").
• Tests serve as a way to document what has been tested and what hasn't.

When to write tests

Tests are written throughout the entire life cycle of a project (e.g. when a new feature is being implemented, when new details of an implementation become known, or when a bug has been fixed). Having that said, we strongly prefer to write the majority of the test cases before -- or at the same time -- a feature is being designed and implemented. This approach has a couple of impactful benefits:

• Writing the tests early gives them more weight. Tests are first class citizens in our workflow.
• Testing early ensures that the code that is produced is, not only testable, but optimized for testing.
• When the implementation is done you're actually done. As opposed to finishing a feature and then having another chore awaiting your attention immediately.
• If you do not start with a test that fails you'll need to "break" the feature to verify that the test works when you do write it.
• It eliminates some false positive tests (e.g. tests that are written just to pass for an existing feature).
• It offers some protection against management abuse. If you ship a working feature without tests it can be very tempting to move on to the next feature.
• Writing the tests early forces you to consider design early and allows you to think things through. You'll have to consider the desired input and output, and how the feature should actually behave.
• No big rush at the end of the project where all of the bugs are exposed at once when the tests are added.
• Early testing enables some fun and efficient ways to collaborate in pair-programming exercises.

Different types of tests

There are a lot of different types of tests that can be used (anything from monkey testing to exploratory testing to black box testing ). These are the types of tests we run on a regular basis:

More frequently run

Unit tests : Low level tests that verify individual functions, methods of classes, or modules. Example : A test that verifies that a function that converts Kelvin to Celsius is correct.

Integration tests : Tests that verify different modules or services and how they work together. Example : A test that verifies that an endpoint can validate and store data correctly in the database.

End-to-end tests : Tests that replicate user behaviour in a more complete environment. Example : A test that uploads a file to an ingestion server, awaits an email confirmation when the data has been processed, and then verifies the output.

Regression tests : Tests that are run when new features are added, bugs are fixed, dependencies are updated etc. Regression tests are typically a subset of the end-to-end, functional integration, and unit tests.

Less frequently run

Performance tests : Tests that evaluate how the system performs under certain workloads. Example : A test that verifies that a slice of data can be retrieved in a timely fashion even when there is 2TB of data in the database.

UI/UX and accessibility tests : Tests that verify whether the user interfaces work as intended on the target platforms and that the level of usability is acceptable. We also verify that the interface is accessible to people with disabilities.

Recovery tests : Tests that verify that the system can recover from disasters such as failed disks or network errors. Example : Verify that a database can be recovered from backups.

Security tests : Tests that verify that the application is secure from internal and external threats. This includes vulnerability and security scanning, code and dependency analysis, penetration testing, authorization validation, and more. We do basic security testing in-house and work with experts for more thorough testing.

General recommendations

• You do not need to test every single line of a code base. Far from it. Start out by focusing on all public features and more complex units of functionality. Expand as the project matures. After a while you’ll be able to intuitively know what to test and what not to test.
• Tests should be easy to run locally. If your application depends on any external service to run it should either be mocked, disabled using a feature-flag, or (preferably) included in the test suite.
• Testing can — and should be — automated. Testing when a feature is pushed ensures that code that breaks existing functionality doesn't make it into the main branch.
• Tests should be isolated and the state should be reset between each and every test. If your project uses a database, make sure a fresh copy is provided for every individual test.
• Tests should be run in an environment that is as similar to the production environment as possible.
• Don't forget negative tests. It is equally important to verify that the application doesn't break when bad data is entered or unexpected things happen.
• Be pragmatic. It is ok to use as many asserts in a single test as your devious little heart desires.
• Make sure you assert the behavior, not the implementation details. A test normally shouldn't fail when you modify the inner workings of the code.
• Keep the tests simple! We don't want to end up having to write tests to test the tests ...
• Use tests as an entry point when debugging. Tests are very useful when you need to identify and isolate an issue, and you'll end up with some nice regression tests when the problem has been fixed.
• 100% code coverage isn't a metric we're striving for, but coverage can be a useful tool to help you find what you've missed.