Install local packages on dask_gateway

[José Gómez-Dans (jgomezdans)]

Hi,
I want to use my own little python package on PC using a dask delayed workflow. If I copy my code into a notebook cell, it works like a charm, but if I install the package on my jupyter machine (pip install -e .), the workers in the dask_gateway cluster can't find parts of the package. If I start a LocalCluster, everything works. I have tried running the pip command on the cluster nodes too, e.g.

import os
def update_environment():
    os.system("pip install -i /home/jovyan/my_pkg/")
    
from dask_gateway import GatewayCluster
cluster = GatewayCluster()  # Creates the Dask Scheduler. Might take a minute.

client = cluster.get_client()
client.run(update_environment)

This doesn't work (fails with message saying it can't find package import). Another option is

from dask.distributed.diagnostics.plugin import PipInstall

plugin = PipInstall(packages=["file://home/jovyan/<blahblah>/"])
client.register_worker_plugin(plugin, name="<blahblah>")

However, while this doesn't flag any errors, and the dashboard looks as if the dask graph has been built, it's just that all tasks are "waiting" forever.

Any hints?

[Tom Augspurger (TomAugspurger)]

You might have a couple options:

1. If your package is just a single python module and doesn't do any fancy package things, you can use UploadFile to distribute the file to the workers: https://distributed.dask.org/en/stable/plugins.html?highlight=UploadFile#distributed.diagnostics.plugin.UploadFile
2. You can put your package somewhere publicly accessible (e.g. GitHub) and use the PipInstall plugin from there
3. Write your own plugin that uploads the file (maybe using UploadFile) and then pip installs it from the worker's local copy.

[José Gómez-Dans (jgomezdans)]

Also, I note that there's an UploadDirectory plugin that you can use that will pack up a whole directory, unpack it in the workers and add them to the path.

from distributed.diagnostics.plugin import UploadDirectory
client.register_worker_plugin(UploadDirectory("/home/jovyan/my_codez/", update_path=True, restart=True), nanny=True)