Efficient Consensus and Synchronization for Distributed Systems

Abstract: Recent interest in decentralized applications calls for distributed systems that replicate their states across a large number of servers communicating over wide-area networks. We propose near-optimal solutions to two fundamental problems in the design and implementation of such systems: consensus and synchronization of system states. First, we propose a universal decomposition of distributed consensus protocols that offloads network-bound tasks from the critical paths, enabling near-optimal throughput and liveness on fluctuating networks. Second, we propose a rateless error-correcting code for reconciling set differences, enabling pairs of servers to synchronize system states with near-optimal communication and computation costs. We theoretically analyze these solutions and implement end-to-end systems to demonstrate strong real-world benefits.