GHSA-w3vc-fx9p-wp4v

Summary

jupyter-server-proxy is used to expose ports local to a Jupyter server listening to web traffic to the Jupyter server's authenticated users by proxying web requests and websockets. Dependent packages (partial list) also use jupyter-server-proxy to expose other popular interactive applications (such as RStudio, Linux Desktop via VNC, Code Server, Panel, etc) along with the Jupyter server. This feature is commonly used in hosted environments (such as a JupyterHub) to expose non-Jupyter interactive frontends or APIs to the user.

jupyter-server-proxy did not check user authentication appropriately when proxying websockets, allowing unauthenticated access to anyone who had network access to the Jupyter server endpoint.

Impact

This vulnerability can allow unauthenticated remote access to any websocket endpoint set up to be accessible via jupyter-server-proxy. In many cases (such as when exposing RStudio via jupyter-rsession-proxy or a remote Linux Desktop / VNC via jupyter-remote-desktop-proxy), this leads to remote unauthenticated arbitrary code execution, due to how they use websockets. The websocket endpoints exposed by jupyter_server itself is not affected. Projects that do not rely on websockets are also not affected.

Remediation

Upgrade jupyter-server-proxy to a patched version and restart any running Jupyter server.

You may not be installing jupyter-server-proxy directly, but have it be pulled in as a dependency (partial list of dependent packages) - so you may be vulnerable even if you aren't directly depending on jupyter-server-proxy.

For JupyterHub admins of TLJH installations

To secure a tljh deployment's user servers, first check if jupyter-server-proxy is installed in the user environment with a vulnerable version. If it is, patch the vulnerability and consider terminating currently running user servers.

1. Check for vulnerability

As an JupyterHub admin from a terminal in a started user server, you can do:

sudo -E python3 -c '
try:
    import jupyter_server_proxy
    is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
    is_vulnerable = False
if is_vulnerable:
    print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
    print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'

Alternatively as a root user on the server where tljh is installed, you can do:

sudo PATH=/opt/tljh/user/bin:${PATH} python3 -c '
try:
    import jupyter_server_proxy
    is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
    is_vulnerable = False
if is_vulnerable:
    print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
    print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'

2. Patch detected vulnerability

As an JupyterHub admin from a terminal in a started user server, you can do:

sudo -E pip install "jupyter-server-proxy>=3.2.3,!=4.0.0,!=4.1.0"

Alternatively as a root user on the server where tljh is installed, you can do:

sudo PATH=/opt/tljh/user/bin:${PATH} pip install "jupyter-server-proxy>=3.2.3,!=4.0.0,!=4.1.0"

3. Consider terminating currently running user servers

User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at https://<your domain>/hub/admin.

For JupyterHub admins of Z2JH installations

To secure your z2jh deployment's user servers, first consider if one or more user environments is or may be vulnerable, then ensure new user servers' aren't started with the vulnerability, and finally consider terminating currently running user servers. The steps below guide you to do so.

1. Check for vulnerabilities

Consider all docker images that user servers' environment may be based on. If your deployment expose a fixed set of images, you may be able to update them to non-vulnerable versions.

To check if an individual docker image is vulnerable, use a command like:

CHECK_IMAGE=jupyter/base-notebook:2023-10-20
docker run --rm $CHECK_IMAGE python3 -c '
try:
    import jupyter_server_proxy
    is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
    is_vulnerable = False
if is_vulnerable:
    print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
    print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'

Note that if you reference an image with a mutable tag, such as quay.io/jupyter/pangeo-notebook:master, you should ensure a new version is used by configuring the image pull policy so that an older vulnerable version isn't kept being used because it was already available on a Kubernetes node.

singleuser:
  image:
    name: quay.io/jupyter/pangeo-notebook
    tag: master
    # pullPolicy (a.k.a. imagePullPolicy in k8s specification) should be
    # declared to Always if you make use of mutable tags
    pullPolicy: Always

2. Patch vulnerabilities dynamically

If your z2jh deployment still may start vulnerable images for users, you could mount a script that checks and patches the vulnerability before the jupyter server starts.

Below is JupyterHub Helm chart configuration that relies on singleuser.extraFiles and singleuser.cmd to mount a script we use as an entrypoint to dynamically check and patch the vulnerability before jupyter server is started.

Unless you change it, the script will attempt to upgrade jupyter-server-proxy to a non-vulnerable version if needed, and error if it needs to and fails. You can adjust this behavior by adjusting the constants UPGRADE_IF_VULNERABLE and ERROR_IF_VULNERABLE inside the script.

singleuser:
  cmd:
    - /mnt/ghsa-w3vc-fx9p-wp4v/check-patch-run
    - jupyterhub-singleuser
  extraFiles:
    ghsa-w3vc-fx9p-wp4v-check-patch-run:
      mountPath: /mnt/ghsa-w3vc-fx9p-wp4v/check-patch-run
      mode: 0755
      stringData: |
        #!/usr/bin/env python3
        """
        This script is designed to check for and conditionally patch GHSA-w3vc-fx9p-wp4v
        in user servers started by a JupyterHub. The script will execute any command
        passed via arguments if provided, allowing it to wrap a user server startup call
        to `jupyterhub-singleuser` for example.

        Use and function of this script can be further discussed in
        https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/3360.

        Script adjustments:
        - UPGRADE_IF_VULNERABLE
        - ERROR_IF_VULNERABLE

        Script patching assumptions:
        - script is run before the jupyter server starts
        - pip is available
        - pip has sufficient filesystem permissions to upgrade jupyter-server-proxy

        Read more at https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.
        """

        import os
        import subprocess
        import sys

        # adjust these to meet vulnerability mitigation needs
        UPGRADE_IF_VULNERABLE = True
        ERROR_IF_VULNERABLE = True


        def check_vuln():
            """
            Checks for the vulnerability by looking to see if __version__ is available
            as it coincides with the patched versions (3.2.3 and 4.1.1).
            """
            try:
                import jupyter_server_proxy

                return False if hasattr(jupyter_server_proxy, "__version__") else True
            except:
                return False


        def get_version_specifier():
            """
            Returns a pip version specifier for use with `--no-deps` meant to do as
            little as possible besides patching the vulnerability and remaining
            functional.
            """
            old = ["jupyter-server-proxy>=3.2.3,<4"]
            new = ["jupyter-server-proxy>=4.1.1,<5", "simpervisor>=1,<2"]

            try:
                if sys.version_info < (3, 8):
                    return old

                from importlib.metadata import version

                jsp_version = version("jupyter-server-proxy")
                if int(jsp_version.split(".")[0]) < 4:
                    return old
            except:
                pass
            return new


        def patch_vuln():
            """
            Attempts to patch the vulnerability by upgrading jupyter-server-proxy using
            pip. Returns True if the patch is applied successfully, otherwise False.
            """
            # attempt upgrade via pip, takes ~4 seconds
            proc = subprocess.run(
                [sys.executable, "-m", "pip", "--version"],
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
            )
            pip_available = proc.returncode == 0
            if pip_available:
                proc = subprocess.run(
                    [sys.executable, "-m", "pip", "install", "--no-deps"]
                    + get_version_specifier()
                )
                if proc.returncode == 0:
                    return True
            return False


        def main():
            if check_vuln():
                warning_or_error = (
                    "ERROR" if ERROR_IF_VULNERABLE and not UPGRADE_IF_VULNERABLE else "WARNING"
                )
                print(
                    f"{warning_or_error}: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see "
                    "https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.",
                    flush=True,
                )
                if warning_or_error == "ERROR":
                    sys.exit(1)

                if UPGRADE_IF_VULNERABLE:
                    print(
                        "INFO: Attempting to upgrade jupyter-server-proxy using pip...",
                        flush=True,
                    )
                    if patch_vuln():
                        print(
                            "INFO: Attempt to upgrade jupyter-server-proxy succeeded!",
                            flush=True,
                        )
                    else:
                        warning_or_error = "ERROR" if ERROR_IF_VULNERABLE else "WARNING"
                        print(
                            f"{warning_or_error}: Attempt to upgrade jupyter-server-proxy failed!",
                            flush=True,
                        )
                        if warning_or_error == "ERROR":
                            sys.exit(1)

            if len(sys.argv) >= 2:
                print("INFO: Executing provided command", flush=True)
                os.execvp(sys.argv[1], sys.argv[1:])
            else:
                print("INFO: No command to execute provided", flush=True)


        main()

3. Consider terminating currently running user servers

User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at https://<your domain>/hub/admin.

Simple Reproduction

Setup application to proxy

Make a trivial tornado app that has both websocket and regular HTTP endpoints.

from tornado import websocket, web, ioloop

class EchoWebSocket(websocket.WebSocketHandler):
    def open(self):
        print("WebSocket opened")

    def on_message(self, message):
        self.write_message(u"You said: " + message)

    def on_close(self):
        print("WebSocket closed")

class HiHandler(web.RequestHandler):
    def get(self):
        self.write("Hi")

app = web.Application([
    (r'/ws', EchoWebSocket),
    (r'/hi', HiHandler)
])

if __name__ == '__main__':
    app.listen(9500)
    ioloop.IOLoop.instance().start()

Setup a clean environment with jupyter-server-proxy and start a jupyter server instance

We don't need jupyterlab or anything else here, just jupyter-server-proxy would do.

python -m venv clean-env/
source clean-env/bin/activate
pip install jupyter-server-proxy
jupyter server

Verify HTTP requests require authentication

curl -L http://127.0.0.1:8888/proxy/9500/hi

This does not return the Hi response, as expected. Instead, you get the HTML response asking for a token.

This is secure as intended.

Verify websocket requests doesn't authentication

The example makes use of websocat to test websockets. You can use any other tool you are familiar with too.

websocat ws://localhost:8888/proxy/9500/ws

At the terminal, type 'Just testing' and press Enter. You'll get You said: Just testing without any authentication required.