POC of different destinations for rpc and fetch

[penalosa]

TLDR: This is a quick POC of a way to point RPC calls and HTTP requests to different network addresses.

The problem

Wrangler/Miniflare uses a workerd ExternalServer to represent service bindings between Workers in local development. This is because each Worker runs in a different workerd process configured by wrangler dev (ignoring the experimental multiworker support we've been landing recently). The caller will configure:

external: {
	address: address,
	http: {
		style: HttpOptions_Style.PROXY,
	},
},

as the service binding, where address is a direct address to a capnpConnectHost configured Socket from the callee.

This means that all communication over that service binding will go to the service linked to the capnpConnectHost Socket on the callee workerd.

However, for Workers Assets this presents a problem. Communication over service bindings needs to go to a different Worker depending on whether it's RPC or HTTP (to the user worker and router worker respectively). We need a way to connect to different addresses (Miniflare exposes a Socket for every configured worker) based on the type of communication.

The proposed solution

This PR implements (with bugs, caveats, and build failures) a proposed solution which adds an additional config property to ExternalServer, addressFetch. When workerd is making a connection to an ExternalServer, it should use the address specified in addressFetch for HTTP communication and the address specified in address for RPC communication. If addressFetch is not specified it should use address for both RPC and HTTP communication.

Wrangler/Miniflare can then leverage this by pointing address at the user worker's Socket, and addressFetch at the router worker's Socket

[github-actions]

The generated output of @cloudflare/workers-types has been changed by this PR. If this is intentional, run bazel build //types && rm -rf types/generated-snapshot && cp -r bazel-bin/types/definitions types/generated-snapshot to update the snapshot. Alternatively, you can download the full generated types:

Full Type Diff

[danlapid]

comment required, and I think you moved the comment of the above field?

[danlapid]

We'll need a test as well

[danlapid]

Can't we do the differentiation in the proxy worker instead?

[penalosa]

Can't we do the differentiation in the proxy worker instead?

I'm guessing this means Wrangler's proxy worker? As far as I know, there's no way to forward on RPC requests from a worker. The proxy worker would have to have logic along the lines of "if HTTP send to worker a, if RPC send to worker b", but I don't know where that could be written. It couldn't go in fetch() {} since that would miss RPC methods, and I don't think there's a good way to define a catchall RPC method (AFAIK it's based on the methods available on the prototype, so exporting a Proxy as a WorkerEntrypoint wouldn't work)

[danlapid]

Can't we do the differentiation in the proxy worker instead?

I'm guessing this means Wrangler's proxy worker? As far as I know, there's no way to forward on RPC requests from a worker. The proxy worker would have to have logic along the lines of "if HTTP send to worker a, if RPC send to worker b", but I don't know where that could be written. It couldn't go in fetch() {} since that would miss RPC methods, and I don't think there's a good way to define a catchall RPC method (AFAIK it's based on the methods available on the prototype, so exporting a Proxy as a WorkerEntrypoint wouldn't work)

Are you certain a proxy wouldn't work?
I think you would only need to define the entry point and a proxy might work
I remember taking to Rahul about it and he seemed to believe it's possible

[penalosa]

@danlapid Fairly sure—I think this codepath would fail:

workerd/src/workerd/io/worker.c++

Lines 1920 to 1950 in 9e915ed

	auto protoObj = JSG_REQUIRE_NONNULL(proto.tryCast<jsg::JsObject>(),
	TypeError, "Exported entrypoint class's prototype chain does not end in Object.");
	if (protoObj == prototypeOfObject) {
	// Reached the prototype for `Object`. Stop here.
	break;
	}
	// Awkwardly, the prototype's members are not typically enumerable, so we have to
	// enumerate them rather directly.
	jsg::JsArray properties = protoObj.getPropertyNames(
	js, jsg::KeyCollectionFilter::OWN_ONLY,
	jsg::PropertyFilter::SKIP_SYMBOLS,
	jsg::IndexFilter::SKIP_INDICES);
	for (auto i: kj::zeroTo(properties.size())) {
	auto name = properties.get(js, i).toString(js);
	if (name == "constructor"_kj) {
	// Don't treat special method `constructor` as an exported handler.
	continue;
	}
	// Only report each method name once, even if it overrides a method in a superclass.
	bool isNew = true;
	kj::StringPtr namePtr = seenNames.upsert(kj::mv(name), [&](auto&, auto&&) {
	isNew = false;
	});
	if (isNew) {
	errorReporter.addHandler(entrypointName, namePtr);
	}
	}
	proto = protoObj.getPrototype();

I can definitely give it a quick go though

[penalosa]

@danlapid Having tried it, I can't figure out how to make a Proxy work—not to say it can't, but I can't figure out how. Additionally, I'm not sure doing this in userland is really the right solution here—it feels like this should be a runtime concern (as it is when deployed). If we do it in userland, then anyone else who comes along and wants to build something similar to Workers Assets needs to copy a fairly convoluted proxy worker, compared to just adding a single config property.