High-Speed Networking: Mastering AWS
MTU – The Agreed Weight Limit on the Internet
In this series, we're going to talk about how to make your network work better and faster in AWS. We'll cover different topics that are important for understanding and improving how your network in AWS works.
First, we're starting with something called MTU, or Maximum Transmission Unit. It's a small but important part of your network that helps decide how fast data moves. We'll show you why it's important and how you can set it up the right way to make your network faster.
Whether you're already good at using AWS or just starting out, these posts will have helpful tips and easy steps to follow. So, let's begin this journey together and learn how to speed up your AWS network!
Story
MetroPacket Couriers is a well-known delivery service in the city, renowned for its prompt and secure delivery of packages. Alex Johnson, a frequent user of MetroPacket, needs to send a large package to Sam Lee, who lives on the other side of the city.
Understanding MetroPacket Couriers' Delivery Process
MetroPacket Couriers follows a set of simple yet crucial rules to ensure smooth and efficient package delivery:
Weight Limit for Each Package: When you want to send a package, MetroPacket requires that it weighs no more than 1500 KG. This limit is set to ensure that each package can be transported efficiently and safely.
Handling Oversized Packages: If your package is heavier than 1500 KG, MetroPacket asks that you divide it into smaller parcels. Each of these smaller parcels should weigh a maximum of 1500 KG. This helps in managing the load more effectively and ensures that all packages adhere to the standard weight limit.
MTU – The Standard Load Limit of the Internet
Imagine the internet as a colossal delivery company, one that spans the entire globe, tasked with sending countless packages (data packets) from sender to receiver every second. This immense network, much like a well-oiled machine, operates under a set of rules to ensure every 'delivery' is efficient, timely, and secure.
One of the key rules in this global delivery system is the MTU, or Maximum Transmission Unit. Think of the MTU as the standard load limit for each package the internet can handle. Just like a delivery company that sets a weight limit for each parcel to ensure safe and efficient transport, the internet has an MTU to regulate the size of data packets it transmits.
The widely accepted standard MTU size is 1500 bytes. This is like saying that the internet's delivery trucks (data packets) are designed to carry a load of up to 1500 bytes. This size is chosen to balance efficiency and reliability in data transmission across diverse network infrastructures.
When you send information over the internet, your data is packaged into these 'trucks'. If your data exceeds the MTU limit, just like an oversized parcel, it needs to be divided. This division, known as fragmentation, ensures that each smaller packet adheres to the 1500-byte limit, facilitating smooth transit through the network's roads and highways.
The internet's adherence to the MTU standard is crucial for preventing issues like traffic congestion (network delays) and package loss (data loss). By maintaining this standard load limit, the internet, as a global delivery entity, ensures that your data reaches its destination intact and on time, no matter where in the world it needs to go.
When you're sending a package with MetroPacket Couriers, think of the total weight limit of 1500 KG (or bytes). But, remember, this isn't just the weight of your package alone. MetroPacket wraps your package in their own packaging for safe transport, which adds about 40 KG (or bytes) to the total. So, in the world of data, when we say the MTU is 1500 bytes, this includes both your original data and the additional 'packaging' added by various network layers. Always consider this combined total to ensure your data packets adhere to the network's MTU standards.
However, it's important to note that 1500 bytes is the maximum limit. Not every network, router, or 'road' might support this full capacity. Some might only handle smaller 'trucks,' meaning they can only support smaller data packets. In such cases, the data must be packaged into even smaller sizes than 1500 bytes to ensure it can travel smoothly across these networks.
Seeing the MTU in Action: A Terminal Guide
When working with networks, it's often necessary to know the Maximum Transmission Unit (MTU) - the largest size a data packet can be for transmission. Let's explore how to determine the MTU using tools available in the terminal.
We'll use ICMP (Internet Control Message Protocol) for this task. ICMP is used by network devices to communicate issues, including those related to MTU size.
The ping command is a simple yet powerful tool for this purpose. By sending packets of varying sizes to a destination, we can determine the MTU.
Finding the MTU which is supported by your interface
To discover the MTU value for your network interface, execute the following commands based on your operating system:
For macOS, use:
ifconfig en0For Linux, use:
ip -d link list
These commands will display the MTU setting for your network interface.
It means that the largest size of a single data packet that can be transmitted over my network is 1500 bytes. This is a standard MTU size for many Ethernet networks.
The MTU size of 1500 bytes on a Network Interface Card is not a fixed limit and can be adjusted through specific commands. Here's how you can change the MTU for different operating systems:
Windows:
Open Command Prompt as an administrator.
Use the following command, replacing
Local Area Connectionwith the name of your network interface andXXXXwith the desired MTU size:
netsh interface ipv4 set subinterface "Local Area Connection" mtu=XXXX store=persistentmacOS:
Open Terminal.
Use the following command, replacing
en0with your network interface name andXXXXwith the desired MTU size:
sudo ifconfig en0 mtu XXXXLinux:
Open a terminal.
Use the following command, replacing
eth0with your network interface name andXXXXwith the desired MTU size:
sudo ip link set eth0 mtu XXXXRemember to replace XXXX with the MTU size you want to set. It's important to choose an MTU size that is supported by your network equipment and infrastructure. Setting an MTU size that is too large can cause network issues, while too small a size can lead to inefficient data transmission.
But… Your network supports that ?
Determining the maximum MTU size that your network can support is crucial to ensure optimal performance and avoid potential network issues. You can identify the maximum supported MTU through a process known as "MTU discovery." Here's how you can perform this for different operating systems:
Windows:
Open Command Prompt.
Use the
pingcommand with the-f(do not fragment) and-l(size) options. Start with a high MTU value and decrease it until the packets are no longer fragmented. For example:
ping [destination] -f -l 1472Replace
[destination]with the IP address/Domain you want to test like google.comThe number
1472is the packet size. The MTU is this number plus 28 bytes for the IP and ICMP headers, so if 1472 works, the MTU is 1500.
macOS and Linux:
Open Terminal.
Similar to Windows, use the
pingcommand, but the syntax is slightly different. Start with a high MTU value and decrease it until you find the highest value that doesn't fragment. For example:
ping -D -s 1472 [destination]Replace
[destination]with the IP address you want to test.Here,
-Dsets the 'do not fragment' flag, and-ssets the packet size. If 1472 works, the MTU is 1500.
For all systems, if the packet gets fragmented or lost, reduce the packet size and try again. The idea is to find the largest packet size that can pass through the network without fragmentation. The optimal MTU is this size plus the size of the IP header (20 bytes) and the ICMP header (8 bytes), totaling 28 bytes.
As you can see from the test, sending a packet of 1424 bytes to amazon.com using the command ping -D -s 1424 amazon.com was successful, indicating there were no issues with packets of this size. Next, I'm going to try sending larger packet to see what will happen.
ping -D -s 1600 amazon.com
Interestingly, when I attempted to send a larger packet size to amazon.com, specifically using the command ping -D -s 1600 amazon.com, I encountered a 'Message Too Long' error. This indicates that even if I increase my MTU setting to 1600 bytes, the network won’t accept such large packets.
But here's a curious point: why aren't these oversized packets being fragmented?
The reason lies in the -D flag used in the ping command, which effectively sets the 'Do Not Fragment' bit in the packet. This prevents the packet from being fragmented, leading to the 'Message Too Long' error if the packet exceeds the network's MTU limit.
To allow fragmentation and further explore the network's behavior with larger packet sizes, we need to remove the -D flag from the command. Without this flag, the network can fragment packets that exceed its MTU limit, allowing us to observe how it handles larger packet sizes. The modified command would look like this:
ping -s 1600 amazon.com
Success! After modifying the ping command to allow fragmentation, it worked smoothly. What's happening here is quite interesting. When I attempt to send a packet of 1600 bytes to amazon.com, it's too large for the network's MTU limit. This size might be exceeding the limit of my router, another router along the path, or possibly amazon.com's own network restrictions.
Originally, my Network Interface Card (NIC) was instructed not to fragment packets, leading to the 'Message Too Long' error when exceeding the MTU limit. However, after removing the 'Do Not Fragment' flag, my NIC adopts a different strategy. It acknowledges that the packet size is too large and decides to split the data into smaller, acceptable packets — in this case, 1452 bytes each.
By breaking down the original 1600-byte packet into smaller chunks of 1452 bytes, my NIC ensures that the data conforms to the network's MTU limit and can be successfully transmitted. This process demonstrates the adaptability of network protocols in handling data transmission over various network conditions and constraints.
Now that we have a basic understanding of MTU and its implications, let's delve into how we can leverage this knowledge to boost network performance in AWS (Amazon Web Services).
For more detailed insights on MTU Discovery, please visit the provided link.
But before we dive into the specifics of AWS MTU settings, it's essential to shed some light on a key concept: JUMBO FRAMES! Understanding jumbo frames is crucial as they play a significant role in network performance, particularly when dealing with AWS's advanced network infrastructure
JUMBO FRAMES > 1500
When we talk about Jumbo Frames, we're referring to Ethernet frames with a payload greater than the standard 1500 bytes used in traditional networks. These larger frames are pivotal in certain network environments, especially when considering the optimization of AWS network performance.
Benefits;
Reduced overhead, such as the number of frames sent and the associated CPU load for processing these frames.
Enhanced network throughput and decrease latency
Happy services, happy SRE, happy networks…
Go on with out Story
While it might be acceptable for Alex to use standard 1500 kg packages for sending occasional parcels to Sam in their personal life, businesses often have different requirements, especially when dealing with high-volume package shipments. In this case, two companies, GlamourGlow Cosmetics, a makeup products provider, and ShopNShip Online, an e-commerce giant, have unique demands.
Recognizing the need for more extensive packaging options, GlamourGlow Cosmetics and ShopNShip Online collaborate with MetroPacket Couriers to establish a specialized line of delivery services. This new service is tailored to their specific needs, offering the capability to send larger packets that accommodate makeup products and e-commerce shipments of varying sizes. With this partnership, both companies can now enjoy the benefits of sending jumbo frames, akin to optimizing their delivery processes for faster and more efficient shipments.
With the establishment of the specialized delivery line by MetroPacket Couriers, GlamourGlow Cosmetics and ShopNShip Online have gained a powerful tool to streamline their operations. This dedicated service allows them to send packages of up to 9000 bytes (equivalent to digital kilograms) to each other, unlocking a world of possibilities.
However, these expanded capabilities come with certain conditions and considerations. GlamourGlow Cosmetics and ShopNShip Online must ensure that their systems and packaging procedures align with the new standards set by MetroPacket Couriers. It involves optimizing their internal processes to handle larger payloads efficiently and securely.
AWS and Jumbo Frames
VPC
Jumbo frames are supported in the VPC by default, so you don’t need to do anything. And this value is up to 9001 MTU. However, traffic is limited to a maximum MTU of 1500 in the following cases:
Traffic over an internet gateway
Traffic over an inter-region VPC peering connection
Traffic over VPN connections
Traffic outside of a given AWS Region for EC2-Classic
Jumbo Frames are also supported between VPC and On-Premise network by using AWS DirectConnect.
Caution: If your packets are leaving the VPC (to the internet), your packets will be fragmented to 1500 bytes (or whatever the MTU is supported), and if somehow you have enabled the “don’t fragment” flag, your packages;
Will return back, if your ICMP is enabled to tell fragment yourself
Will drop, if your ICMP is disabled.
AWS EC2 Placement Groups
Using Jumbo Frames in the EC2 Placement group is going to provide you the maximum throughput.
Instance Types
All Amazon EC2 instance types support 1500 MTU and all current generation instance types support jumbo frames. The following previous generation instance types support jumbo frames: A1, C3, G2, I2, M3, and R3.
To check path MTU using tracepath
Use the following command to check the path MTU between your EC2 instance and another host. You can use a DNS name or an IP address as the destination. If the destination is another EC2 instance, verify that the security group allows inbound UDP traffic. This example checks the path MTU between an EC2 instance and amazon.com.
[ec2-user ~]$ tracepath amazon.com
1?: [LOCALHOST] pmtu 9001
1: ip-172-31-16-1.us-west-1.compute.internal (172.31.16.1) 0.187ms pmtu 1500
1: no reply
2: no reply
3: no reply
4: 100.64.16.241 (100.64.16.241) 0.574ms
5: 72.21.222.221 (72.21.222.221) 84.447ms asymm 21
6: 205.251.229.97 (205.251.229.97) 79.970ms asymm 19
7: 72.21.222.194 (72.21.222.194) 96.546ms asymm 16
8: 72.21.222.239 (72.21.222.239) 79.244ms asymm 15
9: 205.251.225.73 (205.251.225.73) 91.867ms asymm 16
...
31: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
In this example, the path MTU is 1500.
Check and set the MTU on your Linux instance
Some instances are configured to use jumbo frames, and others are configured to use standard frame sizes. You may want to use jumbo frames for network traffic within your VPC or you may want to use standard frames for internet traffic. Whatever your use case, it’s recommended verifying that your instance will behave the way you expect it to. You can use the procedures in this section to check your network interface's MTU setting and modify it if needed.
To check the MTU setting on a Linux instance
You can check the current MTU value using the following ip command. Note that in the example output, mtu 9001 indicates that this instance uses jumbo frames.
[ec2-user ~]$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 02:90:c0:b7:9e:d1 brd ff:ff:ff:ff:ff:ffTo set the MTU value on a Linux instance
You can set the MTU value using the ip command. The following command sets the desired MTU value to 1500, but you could use 9001 instead.
[ec2-user ~]$ sudo ip link set dev eth0 mtu 1500(Optional) To persist your network MTU setting after a reboot, modify the following configuration files, based on your operating system type.
For Amazon Linux 2, add the following line to the
/etc/sysconfig/network-scripts/ifcfg-eth0file:
MTU=1500Add the following line to the
/etc/dhcp/dhclient.conffile:
request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers;For Amazon Linux, add the following lines to your
/etc/dhcp/dhclient-eth0.conffile.
interface "eth0" {
supersede interface-mtu 1500;
}For other Linux distributions, consult their specific documentation.
(Optional) Reboot your instance and verify that the MTU setting is correct.
Here's a concise summary of different MTU sizes in AWS:
Within the VPC: You can use larger frames, supporting up to 9001 bytes for more efficient data transfers within your Virtual Private Cloud (VPC).
Over the VPC Endpoint: When using VPC endpoints, you can send packets of up to 8500 bytes, optimizing data exchange.
Over the IGW (Internet Gateway): For traffic that goes over the internet gateway, the maximum packet size is 1500 bytes, which is the standard.
Intra-Region VPC Peering: When peering within the same AWS region, you can take advantage of larger frames, supporting up to 9001 bytes for enhanced data transfer.
Inter-Region VPC Peering: For peering between different AWS regions, the maximum packet size remains at 1500 bytes.
[On-Premise] Over the VPN with VGW (Virtual Private Gateway): When using a VPN connection with a Virtual Private Gateway, the packet size is limited to 1500 bytes.
[On-Premise] Over the VPN with TGW (Transit Gateway): Similarly, when using a VPN connection with a Transit Gateway, the maximum packet size is 1500 bytes.
[On-Premise] Over Direct Connect: When sending data over AWS Direct Connect, the packet size remains at the standard 1500 bytes.
[On-Premise] Over DX (Direct Connect) via TGW (Transit Gateway): If you're using Direct Connect via Transit Gateway, you can send packets of up to 8500 bytes for more efficient data transfer.
These different packet sizes cater to various AWS network scenarios, optimizing data transmission based on specific requirements.
So, you've now explored the first part of improving your networking in AWS and even between on-premises setups. It's a smart move to grasp the concept of MTU (Maximum Transmission Unit) as it can significantly impact how you design your connections. Whether it's optimizing peering arrangements, leveraging VPC endpoints, or making informed network decisions, understanding MTU opens the door to designing your AWS network with the best options. By doing so, you can ensure smoother and more efficient data transfers, ultimately enhancing your AWS experience.