IM Systems in Depth

The map

The whole system by layer in one picture, with one message’s path numbered. The overview on the home page has four layers and four steps; this is the engineering view, down to every part and every hop.

How one message crosses the system: first it is received safely (steps 1–4), then delivered to every device asynchronously (5–7); a device that was offline pulls it when it returns (8).

Sender: Ana’s phoneStill on Saturday?IM SDK: outbox, local database, syncReceiver: Bena sync cursor per devicePhoneonlineWebofflineInternet: TCP or WebSocket, TLSIM serverBusinesscalled, not passedAccounts, authRelations, rulesblocks, reviewchecked before ACKGroupsmember listsOpen APImsgs, callbacksConnectionholds connections,pushes to devicesgrows with devicesLocatorpicks a gatewayGatewaysone connection per online device: auth, heartbeats, trafficMessagereceives: no loss,no repeats, in ordergrows with messagesMessage serviceone owner per chat: dedup, seqDispatchdelivers toevery devicegrows with fan-outRecipientschats, groupsFan-outwrite or readRoutingfinds gatewaysOnline pushOffline pushSyncStoragemessages, stategrows with retentionMessagesper chatRead cursorsread up toOnline routesdevice → gatewayMetadata, cachechats, filesInboxesone per userInfrastructureMessage queueDiscovery, configRate limitsMetrics, tracing↑ sync↓ asyncMessage logdurable queueoutside the IMBusiness systemsapp backend, supportoutside the IMOS pushAPNs, FCM, vendorsaccounts, relations, messageswakes a phone in the background12345678
  1. Send: the message enters a gateway over Ana’s connection
  2. The gateway reads the conversation ID and forwards it to the message service that owns it
  3. Check whether the message ID was seen recently (a resend gets back its original seq and ACK; a much later resend is stopped by a unique key in storage), then check the sender may post, assign the chat’s next seq, append to the message log (a durable queue)
  4. Once the append succeeds, the message service ACKs: the sender waits only this long
  5. Dispatch reads the log, writes the message store, then delivers (asynchronous from here on)
  6. Work out the recipients, fan out (write fan-out copies into each inbox), look up online routes
  7. Push through the gateway Ben’s phone is on. A phone with no live connection gets an OS push instead, only a hint; the web app uses no OS push here and catches up when opened
  8. A device that was offline comes back with its sync cursor (its position in its own inbox), and the sync service returns what it missed from the inbox (write fan-out) or the group’s timeline (read fan-out); an online device that sees a gap in the chat’s seq pulls too, and overlaps between push and pull are dropped by seq

blue: the message’s pathblue dashed: the pull grey: calls, writes, notificationsdashed box: outside the IMdotted: the sync/async boundary

Many teams say “access, logic, storage”: the connection layer is the access layer, and the message and dispatch layers together are the logic layer. The message log usually runs on the message queue in the infrastructure row, for example as a topic partitioned by conversation.

Point at or tap a layer or card to see its job, why it is hard, and its chapters.