Before this change, connecting a source meant a single blocking request that spun up a warehouse synchronously — ~20s of third-party work in the user's critical path, with no shared notion of progress. This PR turns provisioning into an explicit state machine that runs off the request path: the connection row is created immediately in PROVISIONING, the slow Fivetran/MotherDuck work happens in a durable job, and the client renders state pushed over SSE. Below is the flow end to end — every screen is a real run.
The warehouse is created on demand
Clicking Connect on any source provisions the managed warehouse if the org doesn't have one yet. The row is written immediately in PROVISIONING and the setup tests run out of band, so the user isn't blocked on Fivetran. The header reads “Setting up your warehouse…”.
await prisma.databaseConnection.create({
data: { /* … */ warehouseProvisioningStatus: 'PROVISIONING' },
});
// slow setup tests run in a durable job, not the request:
await getJob(runMotherDuckWarehouseSetupTestsJob).emit(
{ connectionId, userId }, { singletonKey: connectionId },
);Connector-first: the source drives provisioning
The flow is connector-first — you pick a source (here a no-auth Webhooks connector), and the warehouse provisions underneath it. The connector is created against the warehouse's Fivetran group and shows up under Connectors; once setup tests pass the warehouse flips to CONNECTED.
A failed setup is a recoverable state, not a dead end
If Fivetran can't verify the destination, the warehouse moves to FAILED. Instead of the old generic “warehouse unhealthy” banner with no way forward, the homepage now shows a dedicated “Warehouse setup failed” banner with a Retry action that re-runs the setup job.
The retry path is careful about ordering — it enqueues the job before flipping back to PROVISIONING, so a failed enqueue can't strand the warehouse in a permanent loading state.
if (connection.warehouseProvisioningStatus !== 'FAILED') {
throw new Response('Warehouse setup can only be retried after it fails', { status: 400 });
}
try {
await getPgPubSub().notify(`signal:connections:${id}`, {});
await getJob(runMotherDuckWarehouseSetupTestsJob).emit({ connectionId: id, userId }, { singletonKey: id });
} catch (error) {
await prisma.databaseConnection.update({ where: { id }, data: { warehouseProvisioningStatus: 'FAILED' } });
await getPgPubSub().notify(`signal:connections:${id}`, {});
throw error;
}Verified and ready
When the setup tests pass, the warehouse is CONNECTED and the initial sync is enqueued. A successful warehouse is never knocked back to FAILED by an unrelated hiccup — the sync enqueue is isolated so a queue error is logged, not surfaced as a setup failure.
