How to run the tests¶
Software Factory’s functional tests live in the sf-ci repository. You should use the run_tests.sh script as an entry point to run test scenarios.
Deployment test¶
cd sf-ci
./run_tests.sh deploy minimal
This will run the deploy ansible playbook with the minimal architecture of Software Factory. The allinone architecture can be specified too.
The deploy playbook installs the latest development version of Software Factory. This is the recommended way to start with sf-ci. If the deploy scenario fails please notify us directly on IRC or create a bug report on our tracker.
This scenario completes in about 15 minutes.
If you want to use locally built packages then you can prefix the run_tests.sh command with the LOCAL_REPO_PATH=$(pwd)/../zuul-rpm-build:
LOCAL_REPO_PATH=$(pwd)/../zuul-rpm-build ./run_tests.sh deploy minimal
To test small changes, it’s also possible to install the code directly in place, for example:
- sf-config repository content can be rsynced to /usr/share/sf-config
- managesf can be installed using “python setup.py install”
Access to SF’s UI¶
After a successful run of run_tests.sh the UI is accessible via a web browser. The default hostname of a deployment is sftests.com so you should be able to access it using http(s)://sftests.com.
As sftests.com domain might be not resolvable it needs to be added to your host resolver:
echo "<sf-ip> sftests.com" | sudo tee -a /etc/hosts
Local authentication is enabled for the admin user using the password userpass. Some more unprivileged test users are available: user2, user3, user4 with the password userpass.
Please note that the Toogle login form link must be clicked in order to display the login form.
Scratch a deployment¶
To scratch a deployment and start over, use the “–erase” argument:
sudo sfconfig --erase
This command erases all data from the current deployment and uninstalls most of the Software Factory packages. It is recommended to start working on new features or bug fixes on a clean environment.
When switching from a minimal deployment to an allinone it is advised to run that command beforehand to avoid some side effects during functional tests.
Functional tests¶
The functional scenario extends the deploy scenario by:
- Provisioning random data (Git repos, reviews, stories, …)
- Get a backup
- Run health-check playbooks (see sf-ci/health-check/)
- Run functional tests (see sf-ci/tests/functional/)
- Check firefose events
- Erase data (sfconfig –erase)
- Recover the data from the backup (sfconfig –recover)
- Check that provisioned data have been recovered
./run_tests.sh functional allinone
Note that you can use LOCAL_REPO_PATH to include your changes.
This scenario completes in about 60 minutes.
Upgrade test¶
The upgrade scenario simulates an upgrade from the last released version of Software Factory to the current development version.
The scenario runs like this:
- Install and deploy the latest release of Software Factory
- Provision data
- Upgrade the instance to the current development version
- Check the provisioned data
- Run heath-check playbooks
- Run functional tests
./run_tests.sh upgrade allinone
Note that you can use LOCAL_REPO_PATH to include your changes.
This scenario completes in about 60 minutes.
Functional tests¶
After having deployed Software Factory using sf-ci, run:
sudo ./scripts/create_ns.sh nosetests -sv tests/functional/
Most tests can be executed without the create_ns.sh script but some of them require to be wrapped inside a network namespace to simulate external remote access to the Software Factory gateway.
Tips:
- you can use file globs to select specific tests: […]/tests/functional/*zuul*
- -s enables using ‘import pdb; pdb.set_trace()’ within a test
- Within a test insert ‘from nose.tools import set_trace; set_trace()’ to add a breakpoint in nosetests
- –no-byte-compile makes sure no .pyc are run
Health-check playbooks¶
After having deployed Software Factory using sf-ci, run:
sudo ansible-playbook health-check/sf-health-check.yaml
The health-check playbooks complete the functional tests coverage by testing:
- Zuul
- Gerritbot
Testinfra validation¶
After having deployed Software Factory using sf-ci, run:
sudo testinfra /usr/share/sf-config/testinfra
The testinfra checks are simple smoke tests validating Software Factory’s services are up and running.