So you made it all the way to part 3 in our 10Gb Home Area Network Series.

As promised, we are going to overcome the obstacles and expense of networking more than

two systems using a custom built 10Gb switching solution.

If you missed part 1 or part 2 of our 10Gb Home Area Network Series, make sure you go

back and watch those videos first.

Part 3 builds off concepts explained in the first and second videos.

Follow the embedded links now to watch those videos first.

Otherwise, it's time to build a bad-ass 10Gb switch.

Here is a brief overview of what we are going to cover in this video.

Why build a 10Gb switch?

Number one, a custom built switch allows you to meet a specific requirement.

If you are only connecting three systems together, then why pay for an expensive 8 port vendor

switch?

Number two, you can achieve an excellent level of customization.

This solution employs an advanced feature set that you can leverage, if you choose to.

Lastly, for some it just comes down to cash flow.

A custom solution allows you to control your costs better.

Now, the part everyone has been waiting for.

How in the world do you build a custom 10Gb switch?

Well for starters, you have to build a bridge.

According to Wikipedia, a network switch is officially known as a MAC Bridge.

A switch is a multiport network bridge that uses hardware addresses to process and forward

data at the data link layer (layer 2) of the OSI model.

So to summarize, a switch is a bridge is a switch.

So, the plan is to combine multiple Network Interface Cards (NICs) into a reasonable platform,

load an optimized network operating system, and finally, tie multiple ports together using

interface bridging.

So those of you that guessed bridging, you were correct!

Gold star for you!

The end result is a customizable 10Gb switching, and routing, platform that allows for media

conversion.

In other words, you can throw together a dual port 10Gb SFP+ card, a quad port 1Gb copper

card, and a fiber card.

Bridge the ports, and you have just built a media converter.

Now, are you starting to see the possibilities?

There are some pretty interesting things you can do with this type of flexibility.

I'll cover a few scenarios in an upcoming section.

First things first!

Let's talk hardware.

You can do this with a PC or server.

Just like with anything else, your mileage will vary depending on what you use.

See the video description for my setup.

You will need a system with at least two PCIe x8 slots for a 10Gb - 4 Port switch.

Your switch will be limited by the number of PCIe slots that you have, and cards you

use.

Don't forget about your embedded network ports too.

When it comes to the hardware, focus on a motherboard with plenty of bus bandwidth,

maximum PCIe slots (x8 for 10Gb dual cards), and a decent multi-core processor to reduce

CPU loads and increase parallel bandwidth.

With the introduction of technologies like DMA, I/OAT, DCA, and NAPI, PC hardware is

achieving line rate network speeds formally only seen in datacenters on enterprise switches.

Software Defined Networking (SDN) is a prime example of this concept already in play.

For home users, you don't need a heavy-handed solution.

On the other end of the spectrum, aiming for maximum performance, go with a Xeon (E3 or

better) and I/OAT-enabled motherboard when planning for port density.

Now for the most critical component in this entire setup, the network operating system.

And the winner is, VyOS!

What is VyOS?

Why VyOS?

Okay, I'm getting there.

VyOS was originally developed by Brocade Communications and known as Vyatta.

In 2013 they stopped development on Vyatta Core.

A group of enthusiast developers forked the source code and created what is now VyOS.

"VyOS is a Linux-based routing solution built on the Debian Linux distribution, and currently

runs on x86 and x86-64 platforms."

VyOS is an incredibly light-weight, heavily customized, and optimized distro with the

sole purpose of providing Layer 2 and Layer 3 network functionality on a physical or virtual

platform.

Why did I choose VyOS over pfSense (as suggested by viewers)?

The answer is performance, among other things.

To be clear, I have only tested the aforementioned distros.

I'm sure there are other solutions available though.

I discuss further details about my pfSense testing in the summary.

I can personally vouch for VyOS as a routing/switching solution in a production environment.

I know others use it as an edge router in datacenters, and it is widely used in cloud

implementations like Amazon Web Services (AWS).

Awesome!

Now let's dig in with some digital grease, and get these gears moving.

Follow along as I narrate the BIOS Configuration, VyOS Install, Basic Setup, Interface Reorder,

System Setup, Bridge Creation, Bandwidth Testing, and Useful Utilities Demo.

First up, we are going to start by reviewing some important settings in the BIOS.

Check your CPU section and enable: Direct Cache Access (DCA).

Next, under Chipset/Northbridge check for IOAT and VT-d settings.

Enable IOAT for sure and try "No Snoop" and "Relaxed Ordering" if present.

In addition I enabled VT-d and Interrupt Remapping.

Make your way to the PCIe Configuration and look for Above 4G Decoding, Maximum Payload,

and Maximum Read Request.

Maximum Payload and Maximum Read Request are very important and you will see why later

in the video.

Just know that they should be set to the max value.

Save your BIOS settings, and have a bootable copy of VyOS ready.

Boot your system to the VyOS CD.

It's a live CD and allows you to run from boot or install an image to disk or USB.

The default username/password is vyos/vyos.

Run the "install image" command to kick off the install.

You can get context sensitive help at any point in the command line by typing question

mark.

Most of the install process is accepting the defaults.

The install is self-explanatory as you can see.

I installed to a disk drive, but you can install to USB as you only need about 2GB of space total.

Set your new VyOS user password and continue accepting defaults.

When the install completes, make sure to reboot your system without the CD.

Otherwise, any changes you make at this point will be lost.

When you boot into your newly installed image, you are in "Operational Mode".

Take a look at your network interfaces with the "show interfaces" command.

Interface name on the left, and link status on the right.

Capital "D" signifies nothing is connected to our switch at this point.

Plug a copper Cat5 cable into one of your ports and run "show interfaces" again.

Now you can see a lower-case "u" in the right column, signifying the port is up.

A better method is to "blink" the interface you plan to configure.

Run "show interface ethernet eth0 identify" to visually blink the interface you plan to configure.

At this point you should notice the command line is structural in nature.

And remember to use the question mark to better understand your options.

We've been working in Operational Mode.

Now we are going to enter what is known as Configuration Mode.

This allows you to modify your system configuration.

We are going to set the IP address of interface eth1 where we connected our cable.

Notice the slash 24 at the end of the IP address.

This is our subnet mask specified in CIDR notation.

Slash 24 is equivalent to 255.255.255.0.

Now set a description on the interface.

Remember to always label for easier troubleshooting.

Next allow a terminal program like PuTTy to connect to your new interface.

Run the command "compare" to see a list of queued commands ready for execution.

They do not take effect until you issue a "commit" command.

Following the commit, you need to save, so your changes are persistent across reboots.

Did you notice the plus sign next to the queued commands?

This means they are a new addition to your configuration, just as a minus sign would

mean they are being removed from configuration.

Exit Configure Mode and show interfaces to see if changes have taken effect.

In this optional section, I'm going to show how you can reorder your ports however you like.

VyOS may not place your ports in an order that makes sense.

So if you are particular like me, I'll show you how to customize it exactly how you want it.

I'm going to use the Linux ifconfig command to pull, and sort by the MAC address.

Essentially, we are going to reassign the MAC address to the interfaces in a different

order.

Fun fact: The first 3 octets of a MAC address identify the vendor of the hardware.

So try macvendors.com to lookup a NIC vendor.

Awesome troubleshooting technique by the way.

Once you sort by your MAC addresses, copy it out to an editor for reference.

In my list, the bottom 4 MACs belong to 4 embedded Intel NICs.

You can tell by the matching first 5 octets, and last octet order.

So I will make these interfaces eth0 through eth3.

The remaining interfaces, all 10Gb, will be eth4 through eth11 for a total of 12 ports

on my home brew switch.

Before you make this kind of change, it does not hurt to backup your main configuration

file at /config/config.boot.

Enter Configure Mode and use the "hw-id" command to reassign the physical port to the logical

ethernet interface in your preferred order.

Notice I'm using abbreviated versions of commands to minimize typing.

You only have to type enough of the command to make it distinct from other commands.

Here, I'm taking the lower 4 MAC IDs, and assigning them to eth0 through eth3.

Then I assign the top 8 MAC IDs to eth4 through eth11.

Since It's sorted by MAC ID, it will have the effect of correcting the ethernet interface order.

See, how the compare command shows a greater than symbol?

This means you are replacing a configuration item instead of adding or subtracting from

the configuration.

Finally, "commit", "save", "exit", and "reboot" for the re-mapping to take effect.

After rebooting, you may need to relocate your IP address so you can PuTTy back into

your switch.

Here, I show how to remove the configuration.

If you just tried to set the IP again it would add a secondary IP address on the interface.

On the system console, I will remove the IP from eth1 and add it back to eth0 where my

link status is "up".

Now I can PuTTy back in and everything looks good once again.

Connect to your switch with PuTTy, and enter Configure Mode.

If you type "show interfaces" in config mode it will show the current, or running, state

of your configuration.

This is completely different than displaying the contents of your config.boot file.

You can also show other subsections of your running config.

For example "show system", shows the system related aspects of your configuration.

Let's go through some basic configuration settings to get things running smoothly, starting

with your hostname.

Next, set your gateway address so you can communicate with the outside world.

Then finally DNS, and time zone.

Compare your changes, verify, commit and save.

Ping sun.com to test your connectivity to the outside world, and check that time is

syncing correctly with the command "ntpq -p".

You can also use the command "show ntp" from Operational Mode.

This completes the section on System Setup.

Time to build our bridge.

Use a console, or serial connection for this portion, as we need to reconfigure our management

IP again.

Login, show your interfaces, enter Configure Mode, and remove the IP address from eth0.

Notice this time around my delete command stops at "address".

VyOS allows you to remove configuration at various levels.

Now compare, verify, and commit.

Here is where we create our pseudo interface "br0", and assign it an IP address.

This interface represents our bridge, or collection of interfaces forwarding packets to each other

at layer 2.

Assign a meaningful description to your bridge interface.

Looking at our compare statement, notice how default options are assigned to the new bridge.

Commit and Save.

While in Configure Mode you can "show interfaces" by using the keyword "run".

You can now see the newly created interface br0 with assigned IP address, subnet mask

and description.

Let's identify a port where we can connect our 1Gb uplink.

Again, blink eth2 to visually identify.

Okay, plugged in and showing link up on eth2.

Since we have our bridge interface in place, we will assign ethernet ports 2 through 11

to the bridge.

This give us 10 bridged ports.

You could assign all your interfaces to the bridge, but it's not required and you can

change it later anyway.

Remember eth0 through eth3 represent my 1Gb copper ports, while eth4 through eth11 represent

my 10Gb Direct Attach Copper ports.

In the video I mistakenly started with eth3 when I should have started with eth2.

Next add a description to each interface.