scitacean.testing.sftp#
Helpers for running tests with an SFTP server.
This subpackage is primarily meant for testing
scitacean.testing.sftp.SFTPFileTransfer
.
But it can also be used to test downstream code that uses the SFTP file transfer.
The pytest fixtures in this package manage an SFTP server running in a docker container on the local machine. They, therefore, require docker to be installed and running.
Use the scitacean.testing.sftp.fixtures.sftp_fileserver()
fixture to manage the server and use
scitacean.testing.sftp.fixtures.sftp_access()
to get all required access parameters.
See below for examples.
Attention
The fixtures support pytest-xdist but only if all workers run on the local machine (the default).
It may still happen that tests fail due to the complexity of synchronizing start up and shut down of the SFTP server between workers.
See also
Testing user guide.
Examples
In order to test the SFTP file transfer directly, use the provided fixtures to
open a connection manually.
Here, requesting the require_sftp_fileserver
fixture ensures that the server
is running during the test, or that the test gets skipped if SFTP tests are disabled.
Passing the connect
argument as shown ensures that the file transfer
connects to the test server with the correct parameters.
from scitacean.transfer.sftp import SFTPFileTransfer
def test_sftp_upload(
sftp_access,
sftp_connect_with_username_password,
require_sftp_fileserver,
sftp_data_dir,
):
sftp = SFTPFileTransfer(host=sftp_access.host, port=sftp_access.port)
ds = Dataset(...)
with sftp.connect_for_upload(
dataset=ds,
connect=sftp_connect_with_username_password
) as connection:
# do upload
# assert that the file has been copied to sftp_data_dir
Testing the SFTP transfer together with a client requires some additional setup.
See test_client_with_sftp
in
sftp_test.py
for an example.
Implementation notes#
When the server fixture is first used, it initializes the server using these steps:
Create a temporary directory with contents:
tmpdir ├ docker-compose.yaml ├ .env (specifies paths for docker volumes) ├ counter (number of workers currently using the server) ├ counter.lock (file lock) └ data (storage of files) └ seed (populated from scitacean/testing/sftp/sftp_server_seed)
Start docker.
Make data writable by the user in docker. This changes the ownership of data on the host to root (on some machines).
The docker container and its volumes are removed at the end of the tests. The fixture also tries to remove the temporary directory. This can fail as the owner of its contents (in particular data) may have been changed to root. So cleanup can fail and leave the directory behind.
Use the seed directory (sftp_data_dir/"seed"
) to test downloads.
Corresponds to /data/seed
on the server.
Use the base data directory (sftp_data_dir
) to test uploads.
Corresponds to /data
on the server.
The counter and counter.lock files are used to synchronize starting and stopping
of the docker container between processes.
This is required when pytest-xdist
is used.
Otherwise, those files will not be present.
Modules
Pytest fixtures to manage and access a local SFTP server. |