How to run the tests

Software Factory’s functional tests live in the sf-ci repository. You should use the run_tests.sh script as an entry point to run test scenarios.

Deployment test

cd sf-ci
./run_tests.sh deploy minimal

This will run the deploy ansible playbook with the minimal architecture of Software Factory. The allinone architecture can be specified too.

The deploy playbook installs the latest development version of Software Factory. This is the recommended way to start with sf-ci. If the deploy scenario fails please notify us directly on IRC or create a bug report on our tracker.

This scenario completes in about 15 minutes.

If you want to use locally built packages then you can prefix the run_tests.sh command with the LOCAL_REPO_PATH=$(pwd)/../zuul-rpm-build:

LOCAL_REPO_PATH=$(pwd)/../zuul-rpm-build ./run_tests.sh deploy minimal

To test small changes, it’s also possible to install the code directly in place, for example:

  • sf-config repository content can be rsynced to /usr/share/sf-config
  • managesf can be installed using “python setup.py install”

Access to SF’s UI

After a successful run of run_tests.sh the UI is accessible via a web browser. The default hostname of a deployment is sftests.com so you should be able to access it using http(s)://sftests.com.

As sftests.com domain might be not resolvable it needs to be added to your host resolver:

echo "<sf-ip> sftests.com" | sudo tee -a /etc/hosts

Local authentication is enabled for the admin user using the password userpass. Some more unprivileged test users are available: user2, user3, user4 with the password userpass.

Please note that the Toogle login form link must be clicked in order to display the login form.

Scratch a deployment

To scratch a deployment and start over, use the “–erase” argument:

sudo sfconfig --erase

This command erases all data from the current deployment and uninstalls most of the Software Factory packages. It is recommended to start working on new features or bug fixes on a clean environment.

When switching from a minimal deployment to an allinone it is advised to run that command beforehand to avoid some side effects during functional tests.

Functional tests

The functional scenario extends the deploy scenario by:

  • Provisioning random data (Git repos, reviews, stories, …)
  • Get a backup
  • Run health-check playbooks (see sf-ci/health-check/)
  • Run functional tests (see sf-ci/tests/functional/)
  • Check firefose events
  • Erase data (sfconfig –erase)
  • Recover the data from the backup (sfconfig –recover)
  • Check that provisioned data have been recovered
./run_tests.sh functional allinone

Note that you can use LOCAL_REPO_PATH to include your changes.

This scenario completes in about 60 minutes.

Upgrade test

The upgrade scenario simulates an upgrade from the last released version of Software Factory to the current development version.

The scenario runs like this:

  • Install and deploy the latest release of Software Factory
  • Provision data
  • Upgrade the instance to the current development version
  • Check the provisioned data
  • Run heath-check playbooks
  • Run functional tests
./run_tests.sh upgrade allinone

Note that you can use LOCAL_REPO_PATH to include your changes.

This scenario completes in about 60 minutes.

Functional tests

After having deployed Software Factory using sf-ci, run:

sudo ./scripts/create_ns.sh nosetests -sv tests/functional/

Most tests can be executed without the create_ns.sh script but some of them require to be wrapped inside a network namespace to simulate external remote access to the Software Factory gateway.

Tips:

  • you can use file globs to select specific tests: […]/tests/functional/*zuul*
  • -s enables using ‘import pdb; pdb.set_trace()’ within a test
  • Within a test insert ‘from nose.tools import set_trace; set_trace()’ to add a breakpoint in nosetests
  • –no-byte-compile makes sure no .pyc are run

Health-check playbooks

After having deployed Software Factory using sf-ci, run:

sudo ansible-playbook health-check/sf-health-check.yaml

The health-check playbooks complete the functional tests coverage by testing:

  • Zuul
  • Gerritbot

Testinfra validation

After having deployed Software Factory using sf-ci, run:

sudo testinfra /usr/share/sf-config/testinfra

The testinfra checks are simple smoke tests validating Software Factory’s services are up and running.