The Protocol | ZeroTier Documentation (2024)

ZeroTier is a smart programmable Ethernet switch for planet Earth. It allows all networked devices, VMs, containers, and applications to communicate as if they all reside in the same physical data center or cloud region.

Overview​

ZeroTier is a distributed network hypervisor built atop a cryptographically secure global peer to peer network. It provides advanced networkvirtualization and management capabilities on par with an enterprise SDN switch, but across both local and wide area networks and connecting almost any kind of app or device.

This is accomplished by combining a cryptographically addressed and secure peer to peer network (termed VL1) with an Ethernet emulation layer somewhat similar to VXLAN (termed VL2). Our VL2 Ethernet virtualization layer includes advanced enterprise SDN features like fine grained access control rules for network micro-segmentation and security monitoring.

All ZeroTier traffic is encrypted end-to-end using secret keys that only you control. Most traffic flows peer to peer, though we offer free (but slow) relaying for users who cannot establish peer to peer connections.

Everything in the ZeroTier world is controlled by two types of identifier: 40-bit/10-digit ZeroTier addresses and 64-bit/16-digit network IDs. These identifiers are easily distinguished by their length. A ZeroTier address identifies a node or "device" (laptop, phone, server, VM, app, etc.) while a network ID identifies a virtual Ethernet network that can be joined by devices.

ZeroTier addresses can be thought of as port numbers on an enormous planet-wide enterprise Ethernet smart switch supporting VLANs. Network IDs are VLAN IDs to which these ports may be assigned. A single port can be assigned to more than one VLAN.

A ZeroTier address looks like 8056c2e21c and a network ID looks like 8056c2e21c000001. Network IDs are composed of the ZeroTier address of that network's primary controller and an arbitrary 24-bit ID that identifies the network on this controller. Network controllers are roughly analogous to SDN controllers in SDN protocols like OpenFlow, though as with the analogy between VXLAN and VL2 this should not be read to imply that the protocols or design are the same. You can use our convenient and inexpensive SaaS hosted controllers at my.zerotier.com or run your own controller if you don't mind messing around with JSON configuration files or writing scripts to do so.

Origin and Design Philosophy​

The goals and design principles of ZeroTier are inspired by among other things the original Google BeyondCorp paper and the Jericho Forum with its notion of "deperimeterization."

Network Hypervisor​

The ZeroTier network hypervisor (currently found in thenode/subfolder of the ZeroTierOne git repository) is a self-contained networkvirtualization engine that implements an Ethernet virtualization layersimilar to VXLANon top of a global encrypted peer to peer network.

The ZeroTier protocol is original, though aspects of it are similar toVXLAN and IPSec. It has two conceptually separate but closely coupledlayers in the OSI modelsense: VL1 and VL2. VL1 is the underlying peer to peer transportlayer, the "virtual wire," while VL2 is an emulated Ethernet layer thatprovides operating systems and apps with a familiar communicationmedium.

The ZeroTier Peer to Peer Network (VL1)​

A global data center requires a global wire closet.

In conventional networks L1 (OSI layer 1) refers to the actual CAT5/CAT6cables or wireless radio channels over which data is carried and thephysical transceiver chips that modulate and demodulate it. VL1 is apeer to peer network that does the same thing by using encryption,authentication, and a lot of networking tricks to create virtual wireson a dynamic as-needed basis.

Network Topology and Peer Discovery​

VL1 is designed to be zero-configuration. A user can start a newZeroTier node without having to write configuration files or provide theIP addresses of other nodes. It's also designed to be fast. Any twodevices in the world should be able to locate each other and communicatealmost instantly.

To achieve this VL1 is organized like DNS. At the base of the network isa collection of always-present root servers whose role is similar tothat of DNS root nameservers. Roots run thesame software as regular endpoints but reside at fast stable locationson the network and are designated as such by a world definition.World definitions come in two forms: the planet and one or moremoons. The protocol includes a secure mechanism allowing worlddefinitions to be updated in-band if root servers' IP addresses orZeroTier addresses change.

There is only one planet. Earth's root servers are operated by ZeroTier,Inc. as a free service. There are currently four root serversdistributed across the globe and multiple network providers. Almosteveryone in the world has one within less than 100ms network latency fromtheir location.

A node can "orbit" any number of moons. A moon is just a convenient wayto add user-defined root servers to the pool. Users can create moons toreduce dependency on ZeroTier, Inc. infrastructure or to locate rootservers closer for better performance. For on-premise SDN use a clusterof root servers can be located inside a building or data center so thatZeroTier can continue to operate normally if Internet connectivity islost.

Nodes start with no direct links to one another, only upstream to roots(planet and moons). Every peer on VL1 possesses a globally unique 40-bit(10 hex digit) ZeroTier address, but unlike IP addresses these areopaque cryptographic identifiers that encode no routing information. Tocommunicate peers first send packets "up" the tree, and as these packetstraverse the network they trigger the opportunistic creation of directlinks along the way. The tree is constantly trying to "collapse itself"to optimize itself to the pattern of traffic it is carrying.

Peer to peer connection setup goes like this:

1. A wants to send a packet to B, but since it has no direct path itsends it upstream to R (a root).
2. If R has a direct link to B, it forwards the packet there. Otherwiseit sends the packet upstream until planetary roots are reached.Planetary roots know about all nodes, so eventually the packet willreach B if B is online.
3. R also sends a message called rendezvous to A containing hintsabout how it might reach B. Meanwhile the root that forwards thepacket to B sends rendezvous informing B how it might reach A.
4. A and B get their rendezvous messages and attempt to send testmessages to each other, possibly accomplishing holepunching of anyNATs or stateful firewalls that happen to be in the way. If thisworks a direct link is established and packets no longer need totake the scenic route.

Since roots forward packets, A and B can reach each other instantly. Aand B then begin attempting to make a direct peer to peer connection. Ifthis succeeds it results in a faster lower latency link. We call thistransport triggered link provisioning since it's the forwarding of thepacket itself that triggers the peer to peer network to attempt directconnection.

VL1 never gives up. If a direct path can't be established, communicationcan continue through (slower) relaying. Direct connection attemptscontinue forever on a periodic basis. VL1 also has other features forestablishing direct connectivity including LAN peer discovery, portprediction for traversal of symmetric IPv4 NATs, and explicit portmapping using uPnP and/or NAT-PMP if these are available on the localphysical LAN.

A blog post from 2014 by ZeroTier's original author explains some ofthe reasoning behind VL1'sdesign.

Addressing​

Every node is uniquely identified on VL1 by a 40-bit (10 hex digit)ZeroTier address. This address is computed from the public portionof a public/private key pair. A node's address, public key, and privatekey together form its identity.

On devices running ZeroTier One the node identity is stored inidentity.public and identity.secret in the service's homedirectory.

When ZeroTier starts for the first time it generates a new identity. Itthen attempts to advertise it upstream to the network. In the veryunlikely event that the identity's 40-bit unique address is taken, itdiscards it and generates another.

Identities are claimed on a first come first serve basis and currentlyexpire from planetary roots after 60 days of inactivity. If along-dormant device returns it may re-claim its identity unless itsaddress has been taken in the meantime (again, highly unlikely).

The address derivation algorithm used to compute addresses from publickeys imposes a computational cost barrier against the intentionalgeneration of a collision. Currently it would take approximately 10,000CPU-years to do so (assuming e.g. a 3ghz Intel core). This is expensivebut not impossible, but it's only the first line of defense. Aftergenerating a collision an attacker would then have to compromise allupstream nodes, network controllers, and anything else that has recentlycommunicated with the target node and replace their cached identities.

ZeroTier addresses are, once advertised and claimed, a very securemethod of unique identification.

When a node attempts to send a message to another node whose identity isnot cached, it sends a whois query upstream to a root. Roots providean authoritative identity cache.

Cryptography​

If you don't know much about cryptography you can safely skip thissection. TL;DR: packets are end-to-end encrypted and can't be read byroots or anyone else, and we use modern 256-bit crypto in waysrecommended by the professional cryptographers that created it.

Asymmetric public key encryption isCurve25519/Ed25519, a256-bit elliptic curve variant.

Every VL1 packet is encrypted end to end using (as of the currentversion) 256-bitSalsa20 andauthenticated using thePoly1305 messageauthentication (MAC) algorithm. MAC is computed after encryption(encrypt-then-MAC)and the cipher/MAC composition used is identical to the NaCl referenceimplementation.

As of today we do not implement forwardsecrecy or otherstateful cryptographic features in VL1. We don't do this for the sake ofsimplicity, reliability, and code footprint, and because frequentlychanging state makes features like clustering and fail-over much harderto implement. See our discussion onGitHub.

We may implement forward secrecy in the future. For those who want thislevel of security today, we recommend using other cryptographicprotocols such as SSL or SSH over ZeroTier. These protocols typicallyimplement forward secrecy, but using them over ZeroTier also providesthe secondary benefit of defense in depth. Most cryptography iscompromised not by a flaw in encryption but through bugs in theimplementation. If you're using two secure transports, the odds of acritical bug being discovered in both at the same time is very low. TheCPU overhead of double-encryption is not significant for most workloads.

Trusted Paths for Fast Local SDN​

To support the use of ZeroTier as a high performance SDN/NFV protocolover physically secure networks the protocol supports a feature calledtrusted paths. It is possible to configure all ZeroTier devices on agiven network to skip encryption and authentication for traffic over adesignated physical path. This can cut CPU use noticeably in hightraffic scenarios but at the cost of losing virtually all transportsecurity.

Trusted paths do not prevent communication with devices elsewhere, sincetraffic over other paths will be encrypted and authenticated normally.

We don't recommend the use of this feature unless you really need theperformance and you know what you're doing. We also recommend thinkingcarefully before disabling transport security on a cloud privatenetwork. Larger cloud providers such as Amazon and Azure tend to providegood network segregation but many less costly providers offer privatenetworks that are "party lines" and are not much more secure than theopen Internet.

Multipath​

Multipath allows the simultaneous (or conditional) aggregation of multiple physical links into a bond for increased total throughput, load balancing, redundancy, and fault tolerance. There is a set of standard bonding policies available that can be used right out of the box with no configuration. These policies are inspired by the policies offered by the Linux kernel. A bonding policy can be used easily without specifying any additional parameters.

Ethernet Virtualization Layer (VL2)​

VL2 is aVXLAN-likenetwork virtualization protocol with SDN management features. Itimplements secure VLAN boundaries, multicast, rules, capability basedsecurity, and certificate based access control.

VL2 is built atop and carried by VL1, and in so doing it inherits VL1'sencryption and endpoint authentication and can use VL1 asymmetric keysto sign and verify credentials. VL1 also allows us to implement VL2entirely free of concern for underlying physical network topology.Connectivity and routing efficiency issues are VL1 concerns. It'simportant to understand that there is no relationship between VL2virtual networks and VL1 paths. Much like VLAN multiplexing on a wiredLAN, two nodes that share multiple network memberships in common willstill only have one VL1 path (virtual wire) between them.

Network Identifiers and Controllers​

Each VL2 network (VLAN) is identified by a 64-bit (16 hex digit)ZeroTier network ID that contains the 40-bit ZeroTier address of thenetwork's controller and a 24-bit number identifying the network onthe controller.

Network ID: 8056c2e21c123456
 | |
 | Network number on controller
 |
 ZeroTier address of controller

When a node joins a network or requests a network configuration update,it sends a network config query message (via VL1) to the network'scontroller. The controller can then use the node's VL1 address to lookit up on the network and send it the appropriate certificates,credentials, and configuration information. From the perspective of VL2virtual networks, VL1 ZeroTier addresses can be thought of as portnumbers on an enormous global-scale virtual switch.

A common misunderstanding is to conflate network controllers with rootservers (planet and moons). Root servers are connection facilitatorsthat operate at the VL1 level. Network controllers are configurationmanagers and certificate authorities that belong to VL2. Generally rootservers don't join or control virtual networks and network controllersare not root servers, though it is possible to have a node do both.

Controller Security Considerations​

Network controllers serve as certificate authorities for ZeroTiervirtual networks. As such, their identity.secret files should beguarded closely and backed up securely. Compromise of a controller'ssecret key would allow an attacker to issue fraudulent networkconfigurations or admit unauthorized members, while loss of the secretkey results in loss of ability to control the network in any way orissue configuration updates and effectively renders the networkunusable.

It is important that controllers' system clocks remain relativelyaccurate (to within 30-60 seconds) and that they are secure againstremote tampering. Many cloud providers provide secure time sourceseither directly via the hypervisor or via NTP servers within theirnetworks.

Certificates and Other Credentials​

All credentials issued by network controllers to member nodes in a givennetwork are signed by the controller's secret key to allow all networkmembers to verify them. Credentials have timestamp fields populated bythe controller, allowing relative comparison without the need to trustthe node's local system clock.

Credentials are issued only to their owners and are then pushed peer topeer by nodes that wish to communicate with other nodes on the network.This allows networks to grow to enormous sizes without requiring nodesto cache large numbers of credentials or to constantly consult thecontroller.

Credential Types​

• Certificates of Membership: a certificate that a node presentsto obtain the right to communicate on a given network. Certificatesof membership are accepted if they agree, meaning that thesubmitting member's certificate's timestamp differs from therecipient's certificate's timestamp by no more than the recipientcertificate's maximum timestamp delta value. This creates adecentralized moving-window scheme for certificate expirationwithout requiring node clock synchronization or constant checkingwith the controller.
• Revocations: a revocation instantaneously revokes a givencredential by setting a hard timestamp limit before which it willnot be accepted. Revocations are rapidly propagated peer to peeramong members of a network using a rumor mill algorithm, allowing acontroller to revoke a member credential across the entire networkeven if its connection to some members is unreliable.
• Capabilities: a capability is a bundle of network rules that issigned by the controller and can be presented to other members of anetwork to grant the presenter elevated privileges within theframework of the network's base rule set. More on this in thesection on rules.
• Tags: a tag is a key/value pair signed by the controller that isautomatically presented by members to one another and can be matchedon in base or capability network rules. Tags can be used tocategorize members by role, department, classification, etc.
• Certificates of Ownership: these certify that a given networkmember owns something, such as an IP address. These are currentlyonly used to lock down networks against IP address spoofing butcould be used in the future to certify ownership of othernetwork-level entities that can be matched in a filter.

Multicast, ARP, NDP, and Special Addressing Modes​

ZeroTier networks support multicast via a simple publish/subscribesystem.

When a node wishes to receive multicasts for a given multicast group, itadvertises membership in this group to other members of the network withwhich it is communicating and to the network controller. When a nodewishes to send a multicast it both consults its cache of recentadvertisem*nts and periodically solicits additional advertisem*nts.

Broadcast (Ethernet ff:ff:ff:ff:ff:ff) is treated as a multicast groupto which all members subscribe. It can be disabled at the network levelto reduce traffic if it is not needed. IPv4 ARP receives specialhandling (see below) and will still work if normal broadcast isdisabled.

Multicasts are propagated using simple sender-side replication. Thisplaces the full outbound bandwidth load for multicast on the sender andminimizes multicast latency. Network configurations contain anetwork-wide multicast limit configurable at the network controller.This specifies the maximum number of other nodes to which any node willsend a multicast. If the number of known recipients in a given multicastgroup exceeds the multicast limit, the sender chooses a random subset.

There is no global limit on multicast recipients, but setting themulticast limit very high on very large networks could result insignificant bandwidth overhead.

Special Handling of IPv4 ARP Broadcasts​

IPv4 ARP isbuilt on simple Ethernet broadcast and scales poorly on large ordistributed networks. To improve ARP's scalability ZeroTier generates aunique multicast group for each IPv4 address detected on its system andthen transparently intercepts ARP queries and sends them only to thecorrect group. This converts ARP into effectively a unicast or narrowmulticast protocol (like IPv6 NDP) and allows IPv4 ARP to work reliablyacross wide area networks without excess bandwidth consumption. Asimilar strategy is implemented under the hood by a number of enterpriseswitches and WiFi routers designed for deployment on extremely largeLANs. This ARP emulation mode is transparent to the OS and applicationlayers, but it does mean that packet sniffers will not see all ARPqueries on a virtual network the way they typically can on smaller wiredLANs.

Multicast-Free IPv6 Addressing Modes​

IPv6 uses a protocol calledNDP inplace of ARP. It is similar in role and design but uses narrow multicastin place of broadcast for superior scalability on large networks. Thisprotocol nevertheless still imposes the latency of an additionalmulticast lookup whenever a new address is contacted. This can addhundreds of milliseconds over a wide area network, or more if latenciesassociated with pub/sub recipient lookup are significant.

IPv6 addresses are large enough to easily encode ZeroTier addresses. Forfaster operation and better scaling we've implemented several specialIPv6 addressing modes that allow the local node to emulate NDP. Theseare ZeroTier's rfc4193 and 6plane IPv6 address assignmentschemes. If these addressing schemes are enabled on a network, nodeslocally intercept outbound NDP queries for matching addresses and thenlocally generate spoofed NDP replies.

Both modes dramatically reduce initial connection latency betweennetwork members. 6plane additionally exploits NDP emulation totransparently assign an entire IPv6 /80 prefix to every node withoutrequiring any node to possess additional routing table entries. This isdesigned for virtual machine and container hosts that wish toauto-assign IPv6 addresses to guests and is very useful on microservicearchitecture backplane networks.

Finally there is a security benefit to NDP emulation. ZeroTier addressesare cryptographically authenticated, and since Ethernet MAC addresses onnetworks are computed from ZeroTier addresses these are also secure. NDPemulated IPv6 addressing modes are therefore not vulnerable to NDP replyspoofing.

Normal non-NDP-emulated IPv6 addresses (including link-local addresses)can coexist with NDP-emulated addressing schemes. Any NDP queries thatdo not match NDP-emulated addresses are sent via normal multicast.

Ethernet Bridging​

ZeroTier emulates a true Ethernet switch. This includes the ability toL2 bridge other Ethernet networks (wired LAN, WiFi, virtual backplanes,etc.) to virtual networks using conventional Ethernet bridging.

To act as a bridge a network member must be designated as such by thecontroller. This is for security reasons as normal network members arenot permitted to send traffic from any origin other than their MACaddress. Designated bridges also receive special treatment from themulticast algorithm, which more aggressively and directly queries themfor group subscriptions and replicates all broadcast traffic and ARPrequests to them. As a result bridge nodes experience a slightly higheramount of multicast bandwidth overhead.

Bridging has been tested extensively on Linux using the Linux kernelnative bridge, which cleanly handles network MTU mismatch. There arethird party reports of bridging working on other platforms. The detailsof setting up bridging, including how to selectively block traffic likeDHCP that may not be wanted across the bridge, are beyond the scope ofthis manual.

Public Networks​

It is possible to disable access control on a ZeroTier network. A publicnetwork's members do not check certificates of membership, and newmembers to a public network are automatically marked as authorized bytheir host controller. It is not possible to de-authorize a member froma public network.

Rules on the other hand are enforced, so it's possible to implement aspecial purpose public network that only allows access to a few thingsor that only allows a restricted subset of traffic.

Public networks are useful for testing and for peer to peer "partylines" for gaming, chat, and other applications. Participants in publicnetworks are warned to pay special attention to security. If joining apublic network be careful not to expose vulnerable services oraccidentally share private files via open network shares or HTTPservers. Make sure your operating system, applications, and services arefully up to date.

Ad-Hoc Networks​

A special kind of public network called an ad-hoc network may beaccessed by joining a network ID with the format:

ffSSSSEEEE000000
| | | |
| | | Reserved for future use, must be 0
| | End of port range (hex)
| Start of port range (hex)
Reserved ZeroTier address prefix indicating a controller-less network

Ad-hoc networks are public (no access control) networks that have nonetwork controller. Instead their configuration and other credentialsare generated locally. Ad-hoc networks permit only IPv6 UDP and TCPunicast traffic (no multicast or broadcast) using 6plane formatNDP-emulated IPv6 addresses. In addition an ad-hoc network ID encodes anIP port range. UDP packets and TCP SYN (connection open) packets areonly allowed to destination ports within the encoded range.

For example ff00160016000000 is an ad-hoc network allowing only SSH,while ff0000ffff000000 is an ad-hoc network allowing any UDP or TCPport.

Keep in mind that these networks are public and anyone in the entireworld can join them. Care must be taken to avoid exposing vulnerableservices or sharing unwanted files or other resources.