Why they are separate
The separation is by purpose, not by convenience, and the reason is head-of-line blocking. A broadcast to fifty thousand contacts is a large volume of individually unimportant work. A reply to someone who just asked a question is a single piece of work that is urgent by definition — a person is watching a chat window. Put them on one queue and the broadcast is in front, and every reply waits behind it. The same argument separatesflow.execution from messaging.transactional:
inbound processing can be slow, because a flow may make an HTTP call or a retrieval
query. Delivering the reply it produces should not queue behind the next contact’s
inbound message.
sync.external is isolated for a different reason: it depends on systems whose
availability is not ours. When a third party is down and its jobs are retrying with
backoff, that retry storm must not occupy the workers that answer contacts.
Backpressure
Provider rate limits and backpressure are handled preventively, not as a reaction to a provider’s error response. A sender that discovers a limit by being rejected has already spent the request, and on a broadcast that failure multiplies across the whole fan-out — turning one rate limit into tens of thousands of failed jobs, each of which will retry.Which worker takes which
Horizon runs four supervisors, and the grouping is the separation made operational:
The first pair is the latency-critical path — a message arriving and the reply
going out — and it has its own processes so nothing else can crowd it.