Skip to content

Hosted plugin transport

Hosted plugins are read-only v3 sources that run on the customer's machine or in their network and stream observations to Horizon. The transport is generic: the core has no AWS, GCP, Azure, GitHub or Cloudflare logic in it. This page explains the one non-obvious part — why the hosted server speaks both native gRPC and gRPC-Web, and why every self-hosted plugin gets the same path.

The symptom

With native gRPC through a Cloudflare quick tunnel, Preflight and Scan failed with:

rpc error: code = Internal desc = server closed the stream without sending trailers

grpcurl worked against the origin on localhost, and a plain HTTPS request reached the tunnel. The trailers were lost in the middle.

Cloudflare public hostnames do not preserve native gRPC trailers:

  • the quick tunnel defaults to QUIC, and QUIC does not carry gRPC trailers;
  • forcing --protocol http2 still dropped them on the public hostname;
  • a named tunnel with http2Origin: true behaved the same;
  • the named zone also put an interactive challenge in front of API traffic.

So Cloudflare Tunnel is a fine HTTP transport and a bad native-gRPC transport for this use case. The fix is not a provider-specific workaround; it is a protocol the tunnel can carry.

The transport

A hosted source serves two protocols on the same TLS listener:

  • native gRPC (HTTP/2) for a direct customer-runner ingress;
  • gRPC-Web (HTTP/1.1, framed) for the quick tunnel and any HTTP-only edge.

The SDK provides shared/v3/hosted.ServeTLS:

import (
    "github.com/DrSkyle/cloudslash-plugin-sdk/shared/v3"
    v3hosted "github.com/DrSkyle/cloudslash-plugin-sdk/shared/v3/hosted"
)

server := grpc.NewServer()
v3.RegisterSource(server, source{}, publicKey, pluginName)
if err := v3hosted.ServeTLS(ctx, listener, server, cert); err != nil {
    return err
}

It wraps the gRPC server with grpcweb.WrapServer, serves gRPC-Web requests through the wrapper, and forwards native HTTP/2 requests to grpc.Server.ServeHTTP. HTTP/1.1 is used for gRPC-Web; HTTP/2 is used for native gRPC.

On the SaaS side, the hosted source client sends the framed request to /cloudslash.remote.v3.Source/Preflight and /Scan with:

Content-Type: application/grpc-web+proto
X-Grpc-Web: 1
Authorization: Bearer <short-lived signed token>

The response is a sequence of length-prefixed frames. A frame with the high bit set is the status trailer:

[0x80][length:4][grpc-status: 0, grpc-message: ...]

That is what makes the quick tunnel usable: the status and message travel in the body, not in HTTP trailers the edge may strip.

Scan is server-streaming; the client reads data frames until the trailer and validates every batch before persisting anything.

Why the CLI isolates the quick tunnel

cs plugin host <source> --run starts cloudflared tunnel --url ... with:

  • a temporary HOME, so an existing ~/.cloudflared/cert.pem and named-tunnel credentials cannot attach to the trycloudflare hostname;
  • --protocol http2, so the connector uses HTTP/2 rather than QUIC;
  • --no-tls-verify, because the local origin uses a generated self-signed certificate.

Without the isolated HOME, Cloudflare answers the trycloudflare hostname with 404 even though the connector is registered. This is a local-state bug, not a plugin bug, and it affects any plugin equally.

No Cloudflare account, API token, named tunnel or DNS record is required. The quick tunnel is created per host, registered as grpcs://<random>.trycloudflare.com, and replaced when the command restarts.

End-to-end flow

cs plugin host <source> --run
  -> build cmd/hosted for the installed plugin
  -> start a local TLS server on 127.0.0.1:<port>
  -> start an isolated HTTP/2 quick tunnel
  -> register grpcs://<quick-host> with Horizon
  -> Horizon preflight/scan speaks gRPC-Web
  -> plugin server serves gRPC-Web and native gRPC
  -> batches are validated by the core; no provider logic in core

The private listing is workspace-scoped. Re-running the command replaces the endpoint and release; existing connections must be re-verified because the endpoint or capability schema may have changed.

Scan budgets and progress

A hosted scan is a scan.analyze core job. The plugin declares its budget in the v3 capability:

Limits: v3.Limits{
    MaxBatchBytes: 64 << 10,
    MaxOutputBytes: 64 << 20,
    MaxResources:   50000,
    MaxDurationMS:  900000, // 15 minutes
    MaxConcurrency: 8,
}

The API sets a context deadline from MaxDurationMS, marks the job progress: {"phase":"scanning"}, and passes the context through the gRPC-Web request. When the deadline fires, the plugin receives cancellation and the job records hosted source scan timed out after <n>ms instead of a generic "scan failed". Any other stream error is wrapped with its real cause.

The AWS plugin scans regions four at a time and services six at a time; the collector is mutex-protected. That keeps a full region: all scan inside the budget without creating provider-specific code in core.

The scan row in Horizon opens a detail panel with status, stage, elapsed time, attempts, error code and error message. Running jobs refresh every five seconds. A completed job links to its findings.

Adding a provider

A new provider plugin only changes plugin-owned files:

  1. Scaffold a v3 source: cs plugin create <name> --hosted.
  2. Implement Preflight against the provider's real read API.
  3. Implement Scan; preserve ExpectedEnvelope, increment Sequence, honor ctx, and return an explicit final state.
  4. Declare the capability and connection methods in the plugin.
  5. Serve cmd/hosted with v3hosted.ServeTLS.
  6. Test --manifest, Preflight, a real scan and cancellation.

The GCP plugin uses a project ID and a read-only service account JSON key. It scans Compute Engine instances, disks, addresses, forwarding rules, networks, subnetworks and Cloud Storage buckets. The GitHub plugin uses a read-only token and scans repositories, branches, Actions workflows, Actions secret names, deploy keys, webhooks, collaborators and default-branch protection. Both are read-only, both serve the same hosted entrypoint, and neither adds a line to the core.

Provider setup remains plugin-owned:

  • GCP uses setup_mode: console with the service-account console link; the customer creates the key and pastes it into the reviewed form.
  • GitHub uses setup_mode: console with a prefilled token page.
  • AWS uses setup_mode: return with a prefilled CloudFormation review URL whose stack runs in the customer's account and calls the one-time setup callback.

The core only opens a plugin-supplied HTTPS URL, substitutes the one-time state tokens, and receives a generic connection. It has no IAM, CloudFormation, S3, GCP, GitHub or provider-specific setup code.

Debugging

Probe a local hosted server with a gRPC-Web frame:

printf '\x00\x00\x00\x00\x00' > /tmp/empty.frame
curl -sk --http1.1 -D- \
  -H 'Content-Type: application/grpc-web+proto' \
  -H 'X-Grpc-Web: 1' \
  --data-binary @/tmp/empty.frame \
  https://127.0.0.1:<port>/cloudslash.remote.v3.Source/Preflight

A valid server returns HTTP 200, Content-Type: application/grpc-web+proto, and either a grpc-status header or a body trailer frame. A native grpcurl against the same localhost port exercises the native gRPC path.

If the quick tunnel returns 404 for every path, check for a named-tunnel certificate in ~/.cloudflared; use the CLI's isolated quick tunnel rather than running cloudflared by hand from that home directory.