During a high-traffic event, API responses slow down and some requests time out.
The servers have plenty of CPU and memory headroom, yet users see growing delays.
This behavior is related to the fact that the Internet does not reserve circuits in advance but
shares links packet by packet
.
In this article, we look at the
network core
, which carries packets beyond the edge.
We compare how the Internet's
packet switching
works with the
circuit switching
traditionally used by telephone networks, and explain how countless ISPs are connected.
A router does not send a packet onto the next link as soon as the first bit arrives.
It starts transmitting onto the output link only after it has received and stored the entire packet; this is called
store-and-forward
transmission.
Sending a packet of L bits onto a link with a transmission rate of R bps takes L/R seconds.
If there are N links between source and destination, the end-to-end delay of a single packet is N ร L/R, ignoring queuing and propagation delays.
, that is, a queue.
If the output link is busy sending another packet, a newly arrived packet must wait in the queue, and this waiting time is the
queuing delay
.
The queue sends packets out first-come, first-served, like a
FIFO queue
data structure.
But buffer size is finite, so when more packets arrive while the queue is full, either the new packet or a waiting packet is dropped.
This is
packet loss
, and it is why responses slow down and retransmissions increase during peak traffic.
When you run this code, the buffer size is 3, so C1 and B2 are dropped, and C2 enters the queue only after one packet is transmitted and a slot frees up.
Real routers use much larger buffers and a variety of scheduling policies, but the principle that
A router looks at the destination IP address in the packet header to decide which output link to use.
The table it consults is the
forwarding table
, which maps ranges of destination addresses to output links.
Forwarding tables are not written by hand.
Routers exchange path information using
routing protocols
, and the tables are built automatically from the results.
Distinguishing
forwarding as handing one packet to the next link, and routing as computing the overall path,
will help when you study the network layer later.
The difference between packet switching, which shares a link by queuing packets in arrival order at a router, and circuit switching, which reserves slots in advance and uses a dedicated path
Circuit switching reserves resources along the path before communication begins and uses those resources exclusively until the communication ends.
The traditional telephone network is the classic circuit-switched network.
When you place a call, the switches between caller and callee maintain the connection state and allocate a fixed portion of the link capacity to the call.
That is why a
There are two ways for multiple circuits to share a single link.
FDM
(Frequency-Division Multiplexing)
Divides the frequency spectrum and allocates a band to each circuit. It works the same way radio stations use different frequencies.
TDM
(Time-Division Multiplexing)
Divides time into frames and slots, and each circuit uses its assigned slot in every frame.
In both methods, other users cannot use the allocated resources even during a circuit's
silent periods
, when it is not sending data.
Comparing Packet Switching and Circuit Switching ๐
Suppose several users share a 1 Mbps link.
Each user transmits at 100 kbps when active and is active only 10% of the time.
Method
Users supported
Basis
Circuit switching
10 users
1 Mbps รท 100 kbps, with 100 kbps permanently reserved per user
Packet switching
35 users
The probability that 11 or more are active at once is about 0.0004
If 10 or fewer of the 35 users are active at the same time, the aggregate rate does not exceed 1 Mbps, so there is almost no queuing delay.
Using the binomial distribution, the probability that 11 or more users are active simultaneously is about 0.04%.
In other words,
packet switching can support more than three times as many users on the same link at almost the same quality
.
On the other hand, for services where constant delay matters, such as real-time calls, the delay variation of packet switching becomes a drawback.
That is why Internet telephony and video conferencing services compensate for packet switching's variability with techniques such as jitter buffers and loss concealment.
๐จโ๐ป
If your service's traffic spikes at particular times,
request distribution and retry policies
matter as much as server performance.
If every client retries at the same moment, even more packets pile into queues that are already full and loss grows, so apply exponential backoff that adds random jitter to retry intervals.
Access ISPs that connect homes and businesses cannot by themselves connect every host in the world.
Access ISPs connect to regional ISPs, which in turn connect to
tier-1 ISPs
with global networks, and the lower-tier ISP becomes a
customer
that pays the upper-tier ISP.
Today's Internet adds the following elements to this simple hierarchy.
PoP
(Point of Presence)
A group of routers that an upper-tier ISP places to connect lower-tier ISPs.
Multi-homing
An ISP connects to two or more upper-tier ISPs so that it stays connected even if one fails.
Peering
ISPs at the same tier exchange traffic directly with each other without charging fees.
Large content companies and CDNs connect their own data centers with private networks.
They then connect directly to access ISPs or IXPs, bypassing upper-tier ISPs, to deliver content from locations closer to users.
This reduces what they pay upper-tier ISPs and, from the user's perspective, reduces latency.
A network unit operated independently under its own routing policy like this is called an
Autonomous System
(AS), and each AS has a unique ASN (Cloudflare ยท What is an autonomous system?โ).
We will look at the protocols that exchange path information between ASes in detail when we cover routing later.
.
A router stores a packet and forwards it to the next link according to its forwarding table; if the output link is busy, the packet waits in a queue, and if the queue overflows, the packet is dropped.
Circuit switching guarantees consistent quality, but it holds resources even during idle periods, so fewer users can share the same link.
If responses slow down during events, suspect queuing and loss along the path as well as server resources, and check that your retry policy is not making congestion worse.
In the next article, we calculate the
four types of delay, loss, and throughput
that packets experience, including this queuing delay.