scitacean.testing.sftp#

Helpers for running tests with an SFTP server.

This subpackage is primarily meant for testing scitacean.testing.sftp.SFTPFileTransfer. But it can also be used to test downstream code that uses the SFTP file transfer.

The pytest fixtures in this package manage an SFTP server running in a docker container on the local machine. They, therefore, require docker to be installed and running.

Use the scitacean.testing.sftp.fixtures.sftp_fileserver() fixture to manage the server and use scitacean.testing.sftp.fixtures.sftp_access() to get all required access parameters. See below for examples.

Attention

The fixtures support pytest-xdist but only if all workers run on the local machine (the default).

It may still happen that tests fail due to the complexity of synchronizing start up and shut down of the SFTP server between workers.

See also

Testing user guide.

Examples

In order to test the SFTP file transfer directly, use the provided fixtures to open a connection manually. Here, requesting the require_sftp_fileserver fixture ensures that the server is running during the test, or that the test gets skipped if SFTP tests are disabled. Passing the connect argument as shown ensures that the file transfer connects to the test server with the correct parameters.

from scitacean.transfer.sftp import SFTPFileTransfer

def test_sftp_upload(
    sftp_access,
    sftp_connect_with_username_password,
    require_sftp_fileserver,
    sftp_data_dir,
):
    sftp = SFTPFileTransfer(host=sftp_access.host, port=sftp_access.port)
    ds = Dataset(...)
    with sftp.connect_for_upload(
        dataset=ds,
        connect=sftp_connect_with_username_password
    ) as connection:
        # do upload
    # assert that the file has been copied to sftp_data_dir

Testing the SFTP transfer together with a client requires some additional setup. See test_client_with_sftp in sftp_test.py for an example.

Implementation notes#

When the server fixture is first used, it initializes the server using these steps:

  1. Create a temporary directory with contents:

    tmpdir
      ├ docker-compose.yaml
      ├ .env         (specifies paths for docker volumes)
      ├ counter      (number of workers currently using the server)
      ├ counter.lock (file lock)
      └ data         (storage of files)
         └ seed      (populated from scitacean/testing/sftp/sftp_server_seed)
    
  2. Start docker.

  3. Make data writable by the user in docker. This changes the ownership of data on the host to root (on some machines).

The docker container and its volumes are removed at the end of the tests. The fixture also tries to remove the temporary directory. This can fail as the owner of its contents (in particular data) may have been changed to root. So cleanup can fail and leave the directory behind.

Use the seed directory (sftp_data_dir/"seed") to test downloads. Corresponds to /data/seed on the server.

Use the base data directory (sftp_data_dir) to test uploads. Corresponds to /data on the server.

The counter and counter.lock files are used to synchronize starting and stopping of the docker container between processes. This is required when pytest-xdist is used. Otherwise, those files will not be present.

Modules

scitacean.testing.sftp.fixtures

Pytest fixtures to manage and access a local SFTP server.