def SSHCluster(
hosts: List[str] = None,
connect_options: Union[List[dict], dict] = {},
worker_options: dict = {},
scheduler_options: dict = {},
worker_module: str = "distributed.cli.dask_worker",
remote_python: Union[str, List[str]] = None,
**kwargs,
)
The SSHCluster function deploys a Dask Scheduler and Workers for you on a
set of machine addresses that you provide. The first address will be used
for the scheduler while the rest will be used for the workers (feel free to
repeat the first hostname if you want to have the scheduler and worker
co-habitate one machine.)
You may configure the scheduler and workers by passing
``scheduler_options`` and ``worker_options`` dictionary keywords. See the
``dask.distributed.Scheduler`` and ``dask.distributed.Worker`` classes for
details on the available options, but the defaults should work in most
situations.
You may configure your use of SSH itself using the ``connect_options``
keyword, which passes values to the ``asyncssh.connect`` function. For
more information on these see the documentation for the ``asyncssh``
library https://asyncssh.readthedocs.io .
Parameters
- hosts : List[str]
List of hostnames or addresses on which to launch our cluster.
The first will be used for the scheduler and the rest for workers.
- connect_options : dict or list of dict
Keywords to pass through to :func:`asyncssh.connect`.
This could include things such as ``port``, ``username``, ``password``
or ``known_hosts``. See docs for :func:`asyncssh.connect` and
:class:`asyncssh.SSHClientConnectionOptions` for full information.
If a list it must have the same length as ``hosts``.
- worker_options : dict
Keywords to pass on to workers.
- scheduler_options : dict
Keywords to pass on to scheduler.
- worker_module : str
Python module to call to start the worker.
- remote_python : str or list of str
Path to Python on remote nodes.
Examples
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
... ["localhost", "localhost", "localhost", "localhost"],
... connect_options={"known_hosts": None},
... worker_options={"nthreads": 2},
... scheduler_options={"port": 0, "dashboard_address": ":8797"}
... )
>>> client = Client(cluster)
An example using a different worker module, in particular the
``dask-cuda-worker`` command from the ``dask-cuda`` project.
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
... ["localhost", "hostwithgpus", "anothergpuhost"],
... connect_options={"known_hosts": None},
... scheduler_options={"port": 0, "dashboard_address": ":8797"},
... worker_module='dask_cuda.dask_cuda_worker')
>>> client = Client(cluster)See also
dask.distributed.Scheduler
dask.distributed.Worker
asyncssh.connect
Reexports