Ultimate Guide to Data Center Design Concepts!

[00:00:00] Speaker 1: Hello everyone, welcome to this Spacewalkers live session in which I will present data center design concepts. My name is Abubakar Dungula and I am a network solution architect working for a network business division at Alcatel Lucent Enterprise. So here is the agenda. I will first begin by providing an overview of the de facto architecture used by modern data centers today which is the spinal leaf or clost topology. Then I will discuss the physical architecture design concepts and the requirements such as over subscription ratio and capacity planning. After that I will present the different options to connect your hypervisors to leaf switches including the pros and cons of each option. Next I will present the different layer 2 or layer 3 side designs, layer 2 or layer 3 single side design architectures whether it's virtual chassis based or SPB based. After that I will present the routing options in and out of the data center. Also considering you chose the layer 3 design option, I will present what could be the routing options within your data center. Then we will go through stretching subnets or VLANs within your data center and between different data center sites. After going through single site design options, I will discuss the design options for multi-site data center architectures. And then I will go through the different routing options in multi-site design architectures. Finally, we will look at overlay architecture technologies and considerations to take into account in your underlay network. I will close with some key takeaways. So let's get started. Data centers are not just next generation network upgrades. They signify transformation from an IT centric infrastructure to a service centric infrastructure. But to meet that objective, all components including servers, storage, networks and applications are virtualized to cost effectively deliver computational elasticity and data and application serviceability. So based on business objectives and the type of cloud applications deployed, the key goals of any data center architecture are first of all deterministic latency, then redundancy and high availability, then manageability and lastly scalability. So Alcatel Lucent Enterprise offers a broad range of solutions, each addressing the fundamental requirements of data center networks, both for today and tomorrow. So Spinal Leaf architecture. What is it and why is it the recommended choice for data center architecture? So Spinal Leaf, also called Fatree or Klaus architecture, is a multi-tiered network design named after Charles Klaus. He is the Bell Labs engineer who formalized this architecture for telephone and circuit switch networks. But this architecture is equally applicable to packet switch data communication data center. The main building blocks of the Spinal Leaf architecture are the leaves and the spines. So what are the reasons that makes this architecture so successful? This architecture was developed to overcome the limitations of the traditional hierarchical architecture. With the increase of cloud and containerized infrastructure and the increase of east-west traffic, this topology is the prevalent choice when building a modern data center. So other than providing a non-blocking architecture, Spinal Leaf architecture is a topology of choice for several reasons. First, it has a very simple design. It has simpler cabling and it is very cost effective because it's based on cheaper, compact, fixed, single rack unit switches and it's not based on expensive modular switches. And the cost scales linearly with the number of ports ports and bandwidth is consistent between any two nodes. And if you notice there is a single hop between any pair of leaves, which improves latency and provides a deterministic latency. It also provides horizontal scalability, which makes it easy to scale within a boundary. If you need better over subscription ratio, you can add more uplinks. Or if more ports are required, you can just add more leaves or spine as needed. So the spines, they forward traffic along optimal paths between nodes at layer 2 or layer 3, while the leaves control the flow of traffic between locally connected servers. Cross-sectional interconnect bandwidth can be improved through link aggregation links or by employing layer 3 ECMP. There is a single hop latency for server to server communication within the leaves. Additional latency is a factor when traffic needs to bridge the spine for communication with a maximum of three hops for any to any communication. So how do you design your data center physically? So general design considerations when you design the data center are first, always plan for feature scalability. You need to consider an extra 50% port density than what is required. Second, you have to simplify the design as much as possible. This will allow the network to be easier to troubleshoot and manage. Only add further complexity when it's absolutely required. Third, follow a top-down approach in your design by gathering the application requirements to support with your capacity and bandwidth planning. In terms of the physical design, first you have to choose wire rate switches, especially in the spine nodes. Since most of the traffic is east-west, spine switches need to switch the traffic as fast as possible for interleaf traffic. Based on your application requirements, you can calculate an acceptable over-subscription ratio based on the formula shown here in the slide. When we talk about over-subscription ratio, what we are referring to is the ratio of uplink port capacity to the access port capacity. Typical networks are designed with an over-subscription ratio of 2:1 to 6:1. But in most cases, an acceptable over-subscription ratio is 3:1. This, of course, depends on the traffic patterns of your network, which will help decide on an acceptable over-subscription ratio. Third, you need to calculate the number of access ports required. This is based on the number of physical servers that will be connected to the network. As mentioned previously, always plan for feature scalability. Then we will select the leaf switch model based on the port density requirements. We will calculate the number of leaf switches and the total number of uplinks. This can be calculated based on the above formula. The most common top of rack switch is a 48 port 10-gig edge-facing port and 4 times 40-gig fabric ports uplinks. Finally, we can calculate the number of spine switches. So the minimum number of ports for spine switches should be equal to the number of leaf switches, since each spine will connect to all the leaf switches. As an example mentioned here of the process of designing your physical requirements, assuming an over-subscription ratio of 3:1 and 1,000 port server density. So this is the calculation you can see here. This doesn't consider any extra port density requirements. So let's talk redundant hypervisor attachment options. As you can see illustrated in this slide, there are three options, which I will show in a minute, which are available. The first one is using virtual chassis on your leaf switches along with the link aggregation. The second one is using MCLAG with link aggregation. And the third option is to use NIC teaming with no link aggregation. Let's first discuss the MCLAG option with link aggregation, which is shown in the middle here. So not that other vendors may refer to this feature as MLAG or VPC or SMLT or et cetera. MCLAG only provides the ability to determine LACP aggregates on different leaf nodes and load panels across them. It does not virtualize both nodes from a layer 3 control or management plane point of view, like the virtual chassis does. Both nodes are managed independently and are different entities at layer 3. This introduces issues for DHCP relay, DHCP snooping and multicast and routing and convergence issues. Today, many routers or firewalls are deployed as virtual machines. If so, imagine OSPF hellos being sent as multicast on one of the members of the lag ports. This means that the router will be adjacent to one of the leaf nodes only. In case of leaf node or link failure, the VM will need to rediscover the other leaf node, which will affect convergence time. Next, we will go to the third option, which is to use NIC teaming on your virtual machine. This option can be used when LACP is not supported on your hypervisor or to avoid the additional licensing cost incurred by enabling this feature. Leaf nodes are not virtualized. They are two separate entities with independent control and management plane. The hypervisor's NICs are bonded in active-active mode without LACP. So, traffic is balanced across both interfaces according to the VM's source MAC hash. This means that a given VM will always use the same interface until the interface or the link fails. In such event, the affected VMs will fail over to the remaining interface. For this reason, both interfaces need to be mapped to the same broadcast domain, for example, VLAN or SPB service. Therefore, these broadcast domains need to be extended to both leaf switches. This is accomplished directly by interconnecting both nodes or indirectly by transiting the spine nodes. So, it should be noted here that, except in the case of SPB-based architecture, this option requires PaddingQ protocol since the same broadcast domains are extended across both nodes. Also, in case of opting for overlay protocols such as VXLAN with a hardware VTEP, you will require different VTEP on each leaf switch. And the last option, which is the simplest and best, and the most recommended option in most cases, is to use the virtual chassis with link aggregation. So, LACP provides active-active load balancing and fast failover in the event of a link or a NIC or leaf node failure. Traffic is load balanced according to a hashing function, which can consider the source and destination MAC, IP addresses, or TCP UDP ports. The exact hashing logic is configurable on both the switch and the hypervisor. So, the virtual chassis virtualizes both the leaf nodes from a data control and management point of view and at both layer 2 and layer 3. So, this virtualization brings deployment and operational simplification. In addition, the OmniSwitch side of the link aggregate is automatically configured as soon as it's configured on the hypervisor or server side. So, bringing additional operational simplification. Whilst the virtual chassis feature is not licensed on OmniSwitch product, LACP may be licensed on the hypervisor or the virtualization platform. So, the following points summarize why virtual chassis is the best and recommended option. Virtual chassis is automatic and it provides full node virtualization at both layer 2 and layer 3. It is managed as a single entity with a management plane and allows for in-service software upgrade. It has no issues with DHCP relay and routing appliances. Above all, it's very simple to configure and deploy and operate. So, border nodes connect the data center network to the outside world. For example, the campus network, the one with their internet. Or to services such as firewalling and load balancing. So, there's two options available for external connectivity out of the data center. Here, we are showing the external connectivity out of the data center using border spines. We will cover the next option which is border leaves in the next slide. Using border spines is a good option when there's two spines. And this allows for active-active or active standby redundant external connections or to services. It's also a good option when your north to south traffic is high compared to your east-west traffic. This depends on the traffic patterns from your data center. However, a larger data center may require more than two spines. And using border spines will stop making sense if external connections or services only connected to those two spines. This will result in unbalanced traffic because some spines would receive more traffic than others. In such case, using border leaves may make more sense. So, a trade-off in using border spines is that in case extra bandwidth is required by adding a new spines switch, then it's required to add links to external networks from the new spines switch, which leads to more cost and complexity. The second option is to use border leaves. This is a better option with more east-west traffic. In case bandwidth needs to be added for east-west traffic in the fabric, then additional spines switch can be added without affecting the external connection design. In case one of the external links towards the external router is lost, then only 25% of the bandwidth capacity is lost. So, now we will cover the single-side design options of your data center. The first option is using virtual chassis-based layer 2 architecture. The spine nodes are clustered together in a virtual chassis. Leaf nodes may or may not be clustered in a virtual chassis depending on whether the hypervisors will be attached through the LACP aggregates or not. In this slide, we are showing standalone leaves with the hypervisors connecting without link aggregation. So, virtual chassis-based architecture provides simple active-active and hash-based load balancing at both layer 2 and layer 3. Inter-subnet routing at the virtual chassis, at the spine virtual chassis takes advantage of the virtual chassis high availability and load balancing without any additional latency protocols such as VRRP. There is, however, a six-spine limitation with fewer leaf nodes since the links are required for VFL ports. In this case, where the hypervisors are not attached using LACP link aggregates, then spanning tree protocol is required since broadcast domain is required to be extended. In this case, it should ensure that only the spine nodes are configured as root bridges and root guard feature is enabled on all downlinks. So, here's a virtual chassis-based architecture with virtual chassis leaves and LACP link aggregates. When the downstream nodes such as other switches or hypervisor are multi-homed to all virtual chassis units through a LACP aggregate, node-to-node traffic requires a single hop across the virtual chassis unit and does not need to traverse the VFL or the fabric link. What this means is that traffic is forwarded at wire rate and the VFL is not a bottleneck. So, this is because the traffic forwarding to a LACP member port gives preference to local ports over remote ports across the VFL. This is true for unicast traffic only. BOM traffic may have to be forwarded across the VFL. Now we come to a fully routed architecture based on virtual chassis configured on the leaf nodes. Point-to-point routed interfaces are configured between nodes and you can configure routing protocols such as OSPF, ISIS, or BGP. We will cover single-site routing options in the next session. Hypervisors can be configured with LACP connection to a TOR top-of-rack virtual chassis leaf switch as discussed earlier. Spine switches do not need to be in a virtual chassis. In case using overlay architecture which uses VXLAN, each RACP uses a different VTEP or a virtual tunnel endpoint. Subnet which benefits BOM applications due to hierarchical replication. This will be discussed later in overlay architecture consideration section. Using a lag between the leaf nodes and the spines optimizes the routing table and eliminates the need to forward traffic across the VFL. So in an SPB design with standalone leaves, hypervisors are connected using a NIC teaming which performs per VM load balancing. So there will be as many backbone VLANs as there is spines. And you can use dynamic UNPs and SAPs which will be configured on leaf nodes as per the different customer VLANs on the VMs on the hypervisors. Which will be load balanced across the BVLANs by default. To achieve load balancing on the spines, VRRP is required on the spine nodes and subnets will be split between the spines. There will be as many VRRP groups as there are spines and the priority of each group will be set such that the spine is a master of one group. This will evenly balance the traffic across the BVLANs and the VRRP groups. In case you are not using LACP link aggregate on the hypervisors, you should configure spanning to protocol and you should enable loop back detection on the BAB leaf nodes. So hypervisors can be configured with lag connections to the leaves as per the design shown here. With the leaves configured in the virtual chassis of two leaf switches. SPB by default will only use one of the links to form a virtual chassis connection. It will use one of the links from a virtual chassis connected to separate spines, which is not optimal of course. In order to load balance traffic evenly, links should be bundled in a lag from the spines to the leaves. To avoid unicast traffic forwarding on the VFL, each leaf node should should have a link to each spine. And all the hypervisors should be dual home to every leaf. For routing, SPB and VRRP will be configured as the same as the previous design. So here's a quick summary comparison with pros and cons for layer two versus layer three architectures. Advantages of layer three designs are simple architecture and configuration. And SPB makes it actually much, much easier. It has inherent seamless VM mobility. You can attach multiple hypervisor attachment with link aggregation. It has a fast, fast convergence and many more advantages. In case you're using a virtual chassis based architecture layer two architecture, you have the option. It provides an advantage of active, active load balancing. Of course, if you're not using SPB in your layer two design, then you have to configure spanning two protocol everywhere. And you have to make all the configuration of the root, root bridges as the spines. And you have to enable root guard towards the leaves. And the VLANs need to be configured everywhere across all the switches. And it's susceptible to broadcast storm and everything is a single for their domain. In case you're using a layer three design, of course, it doesn't have the disadvantages of the layer two designs. It has better scalability and stability. And you can use ECMP between your between the leaf nodes towards the spine nodes. Disadvantages, of course, you can use an overlay protocol in case you need VM mobility. VXLAN is one example. You need to have more configuration, since you need to configure the underlay and the overlay. And you need to configure layer three design requires an IP address on each interface, which adds more complexity. And of course, it makes it more difficult to troubleshoot. By forwarding detection, BFD is required for fast convergence. So now let's go over a few routing options within your data center in case you're using a layer three design. The first option is to use OSPF. Using a single area OSPF configuration should satisfy most requirements. Best practice is to configure point-to-point interfaces in the spine leaf links and use ECMP for load balancing. It is better to avoid multi-area configuration and/or summarization as this adds complexity unless it is necessary in your design. In case your fabric gets bigger, rather than implementing multi-area OSPF, it might make more sense to migrate to BGP. In case you're writing on the host, you can implement totally solid areas on your host. This will allow the host to be isolated from the fabric. So let's talk BGP. So many network administrators would be hesitant to use BGP in their data center networks. And for many reasons. However, it should be important to note that very large data center operators such as Microsoft or Facebook, they operate BGP in their data centers. So let's look at a few myths regarding BGP and discuss how these myths can be busted. So myth number one, BGP is a very slow protocol. So the fact is that BGP is a very stable and proven protocol, which is used in the internet. So compared to IGPs, BGP does have a slow convergence process, but this can be tuned to be comparable to IGPs. So myth number two, that BGP is complex and requires a lot of configuration. The fact is that BGP can be simple or complex depending on your routing policies, depending on the routing policies required. BGP configuration can be easily solved with automation. Other advantages of using BGP in your data center is that in case you decide to deploy an overlay protocol such as EVPN or MPLS, you will require BGP anyway. So also you have other advantages is that it supports multiple address families in one protocol. So in case you plan to deploy IPv4 and IPv6. So the second option in terms of the single routing designs is to use IBGP in your data center with a single autonomous system network. Since IBGP requires either full mesh or full mesh FPRs or route reflectors, using route reflectors is more scalable in data center design. The spine switches is the obvious choice to be used as route reflectors. Route reflectors should be in the same route reflectors cluster as we might end up with unsynchronized BGP databases. This can be done by configuring the same cluster ID for the spine. You can also fine tune your BGP parameters. This is recommended in order to optimize your network and speed up the convergence. The first convergence optimization is to enable multipath. This will allow for load balancing. And when multipath support is enabled and the path selection process determines that multiple paths are equal, when the router ID is disregarded, then all the equal paths are installed in the forwarding table. So when multipath is disabled, only the best route entry is installed in the hardware forwarding table. The second optimization is to adjust the advertisement interval. This should also be considered for fast convergence. So this interval is used by the peer to send the BGP update messages to the external peers. So other BGP timers that should be adjusted are keep alive and hold time. The keep alive interval can never be more than one third of the value of the hold time. When the hold time interval is reached without receiving keep alives or other update messages, the peer is considered dead. The last feature that should be enabled is a fast external failover. When this is enabled, this command allows BGP to take immediate action when a directly connected interface on which an external BGP session is established goes down. So normally BGP relies on TCP messages to manage peer connections. FIFO improves upon TCP by resetting the connection as soon as they go down. So since we are only using BGP in this design, we're not using any underlay, only BGP. When the route reflector reflects the routes from the leaf switches, the next hop will not be changed. So once the other leaf switch receives the route, they will not be able to reach the next hop and the route will not be used. So the solution here is to configure, to change the next hop to be the route reflector itself. This is by configuring the next hop itself. This can also be achieved by using route maps, which adds complexity to this design. The third option is to use IBGP in your data center with a single autonomous system, but to use OSPF as an underlay. This is used to advertise the next hop reachability information. This will simplify the topology and you will not require configuration of route maps since the next hop will be reachable once the route is reflected. So this design is the simplest and it allows the possibility to upgrade or remove a switch from the overlay network without disturbing the underlay network. So the same comments apply for the BGP parameters as well. The next option is to use a unique autonomous system number in each switch and use a BGP between the switches. This design does come with certain drawbacks such as path hunting and value routing, which I will discuss. So BGP path hunting, I will show a scenario in the coming slide. It's the process where the BGP routers, they hunt for an alternative path to our route when the primary path becomes unreachable. This means that when our prefix becomes unreachable, BGP will explore longer and longer paths before the prefix disappears from the routing tables everywhere. So path hunting is not only BGP issue, but it's a general distance vector protocol convergence problem since it's based on information received by the peers. So unlike link state protocols where the change is flooded immediately. So and the second issue is that the value routing. So what is value routing? So what is value routing? Assuming that the leaf switches are the valley, traffic does not transit only the spines to reach another leaf, but instead they go through another leaf to reach the destination. So traffic goes in a zigzag pattern. Obviously, this is not optimal in leaf and spine topology as the leaf and spine topology is meant to provide valley free routing. So let's take a scenario to make this concept clearer. So assume we have a route 10.10.10.1/32 connected to leaf one. So in spine one point of view, the route in the BGP table has four different paths as shown in this slide. So the best path is directly to leaf one. Other paths are learned through the other leaf switches two, three and four, but they are not used since BGP only uses the best path. The default advertisement interval for BGP in Omni switch implementation is 30 seconds. What happens when a route 10.10.10.10.1/32 goes down? So an update message with an update message with the route is marked as withdrawn, is sent to all of the peers. So when spine one receives the update message, it withdraws the prefix with the next top as leaf one for that prefix, but it has alternative routes already. So it sends the new update message to leaf one saying it can reach the route via leaf two. At the same time, the withdrawal message sent earlier to spine two is now propagated towards leaf two, three and four. So now leaf switch two sends back an update message to spine one with a withdrawal for this prefix 10.10.10.1/32 since it's not able to reach that route. Spine one BGP table withdraws the route. So when spine one receives the update message, it withdraws the prefix with the next top as leaf two for that prefix, but it has alternative routes. It sends the new update message saying it can reach the route via leaf three. So now leaf three switch sends back an update message to spine one with a withdrawal for the prefix 10.10.10.10.1/32 since it is not able to reach that route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. [00:28:32] Speaker ?: Spine one BGP table withdraws the route. [00:28:33] Speaker 1: Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. [00:28:37] Speaker ?: Spine one BGP table withdraws the route. [00:28:37] Speaker 1: Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine two update message saying it can reach the route via leaf four. So now leaf switch four sends back an update message to spine one with a withdrawal for the prefix since it is not able to reach the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. Spine one BGP table withdraws the route. So as you can see this whole process is called the path hunting. Trying to find the alternative path for this prefix before it's completely removed from the BGP table. So as a reminder, valley free routing. Valley routing, sorry. Traffic does not transit only the spine to reach another leaf, but instead goes through another leaf to reach the destination. So traffic goes in a zigzag pattern. In our scenario here, traffic from leaf 1 to leaf 4 has two routes. Either the blue one or the red one. So let's assume the link, the links shown here are shut down. Now traffic must transit leaf 2 to reach leaf 4. As shown in green here. So obviously this is not optimal for many reasons. So disadvantages for value free routing. There is a longer path to reach the destination and it's unpredictable and undeterministic latency in order to reach the destination. It might take a longer path to reach the destination. And it provides unpredictable paths to the destination and you might have different banded spikes and utilization in your data center. So how do you solve this solution of path hunting? You can use a single and also a value routing. How do you solve them? You can use a single autonomous system in your spine switches. I will show you in the next option. It's coming in the next slide. To solve path hunting, another solution is to use route maps to only advertise locally originated routes. In addition, you can set the advertisement interval to zero, which allows for faster convergence and the path hunting issue will be improved. Another solution to value routing is to use virtual chassis. So you're using a single spine, but you have a six spine limit, of course. So the fifth option, the last design, is to use a single autonomous system in your spine switches with eBGP to your leaf switches. So this solves the path hunting issue since no other paths are installed in the BGP table. At the same time, value routing is eliminated. So now we will speak about stretching subnets across multiple data center sites. So stretching subnets or VLANs is sometimes considered between two data center sites for several reasons. Subnet mobility for disaster recovery purposes. VM mobility used by automatic resource schedules, such as DRS for VMware. Between clusters to optimize compute resources and load balancing. And VM mobility for maintenance purposes. So stretching the layer two domain in general, not only increases the failure domain. It also introduces other drawbacks, such as suboptimal traffic flows, traffic tromboning, and traffic bottlenecks in DCI link. So let's give an example of suboptimal traffic flows, traffic tromboning, and DCI bottlenecks. So let's say if an external user is accessing an application hosted in DCA. Traffic will flow normally in both directions. Let us consider that the VM was moved from DCA to DCB for maintenance purposes. Our DRS determined that it's optimal for resource utilization. Since firewalls and load balancers require session persistence, traffic will still arrive in DCA and traverse the DCI connection to reach the application in DCB. Intersite traffic between different VMs and the same subnet will also traverse the DCI connection. If the application was moved to DCB requires reads or writes to the database, which was original in DCA, this will also traverse the DCI connection. Even if there is a replica of the database in DCB, this will still require primary writing to databases in DCA. For application, application is continuous, access to storage is required, and moving VMs between different sites requires synchronous replication between the sites. This will also traverse a DCI connection. So depending as well on the storage technology, it's possible that DCB storage will be read only and writes needs to be done on DCA storage. In case the application which was moved from DCB needs to communicate with the VM and DCB, traffic flows may suffer from trombone effect, which is shown here. Since the VM requires to communicate with the default gateway, which may still be in DCA, this will cause traffic to traverse the DCI connection to DCA and route back to DCB. However, this can be solved with active-active routing with virtual chassis or active-active VRRP. The DCI link usually does not have enough bandwidth or capacity to handle stretched subnets. And moving a few VMs with a few terabytes of RAM may take hours or days depending on the speed and amount of data moved. And bump traffic will be flooded across both sites using DCI congestion. In conclusion, subnet stretching may look appealing until you analyze the downsides. So this design may no longer be required since clustering no longer requires layer-2 connectivity. So using local subnets for each site is a better design solution for multiple reasons. So stretching subnets is no longer required since clustering no longer requires layer-2 connectivity. And layer-3 clustering across sites is possible. This is achieved using a load balancer. There is better fault isolation, for example, in the event of a broadcast storm, which will not extend to the other site. And traffic flows are optimized and DCI connection will be less congested. And inter-site non-live VM migration can be achieved by changing the DNS increase. This design may be optimal, as highlighted above, but less appealing to the server infrastructure team. A compromised solution is to use local subnets, which can be stretched when required. This solution requires the use of VRRP between both sites. DCA will be the master of its own subnets and DCB will be the master of its own subnets. But the slave interfaces are administratively disabled on the other data center. So the slave interfaces will be enabled only and if required, such as a maintenance or a disaster recovery event. In the event we are required to stretch the subnets and we are using SPB, we will move the VMs from DCA and administratively enable the interfaces in DCB. And the services will self-configure. If we use virtual chassis and lag, it will work in the same way provided we use UNP and MVRP. The same caveats, however, apply in case we are required to move many VMs across the sites in case of a disaster recovery event where the DCI connection will be congested due to the large amounts of data. So now let's discuss multi-site design design options. So when using dark fiber, DWDM or pseudo-wire for DCI connection, LACP link aggregates, if supported, can be configured between both sites. Otherwise, it can be configured statically. So to optimize traffic flow, we can use active-active VRRP. This is achieved by configuring both sites as VRRP master. This can be done by blocking VRRP hellos and ARP requests over the DCI with access control list. ARP requests sent locally will be replied to by the local gateway virtual chassis. To avoid traffic traversing the VFL, we need to attach the hypervisor to both leaves and attach each leaf to all spines. Another option is to only use virtual chassis on the spines, but we will still need to attach each leaf to all spines to avoid traffic traversing the VFL. Both this design and the one in the previous slide will still have this traffic optimization issues mentioned earlier regarding stretched subnets, such as storage and database traffic flows and traffic into and out of the data center. The third option is to use full VC design on the spines across both sites. This will still have the same caveats as before, but also another risk is the split-brain scenario. Split-brain occurs when there's failures on the VFL links, which cause potential MAC and IP duplication. This can be solved by either using out-of-band using EMP remote chassis detection protocol or MBAT using the VC split protocol. In this design, the split-brain scenario can only be detected using out-of-band EMP RCD and not using virtual chassis split protocol since leaf nodes are only local and not connected to the other site. Even if a split-brain scenario is detected, this will cause the entire site to go down. So for SPB-based designs, we cannot use access control list for active-active VRRP by configuring both sides of the VRRP master. This is because traffic across DCI is MAC and MAC encapsulated and ACLs are required to look inside the PBB encapsulation, which is not supported. Traffic could be optimized by load balancing VRRP groups, such as each spine is the master for one group of subnets by using local subnets without stretching. Another option is to use virtual chassis in the lead nodes with the link aggregation as well. So let's discuss a few options when routing out of the data center. So when routing traffic in and out of the data center, then protocols such as OSPF, ISS should not be considered for several reasons. OSPF does not allow for prefix-based policy control. Only metric is the cost and it is set on the interface and not based on the prefix. With firewalls, it can only work as active standby and OSPF requires separate protocol instances for IPv4 and IPv6. OSPF can be acceptable when DCI is very fast and not over long distances. Even with inefficient traffic flows, we will still have plenty of bandwidth and low latency. OSPF can be acceptable if it's within the same building or campus, for example. A better option is to use BGP for the following reasons. BGP allows for granular policy control for prefixes. It allows for better load balancing with firewalls. And it allows better load balancing on a per-prefix basis. The same protocol, MPBGP, it can be used for IPv4 and IPv6 address families. And if SPB is used, whether on the campus or data center site, it's not required to add another routing protocol within the site. So in this design, in this slide, IPGP will be used between the data center sites and eBGP between the data center and the campus. Traffic can be engineered for load balancing using the MED attribute and the local preference, BGP attribute as well, such as some subnets will be advertised with better preference from one data center and with the worst preference on the other to ensure that both links are utilized. So now let's go over a few considerations to make when designing your data center with overlay architecture, such as VXLAN or EVPN. So in an overlay network, data is transmitted over virtual tunnels between nodes. This encapsulation and decapsulation may require an extra overhead, which is one of the considerations that need to be accounted for. So I'm showing an example here of VXLAN overhead. The original layer two frame has a VXLAN header encapsulated, and then this is transmitted across the network within UDP segments carried within IP datagrams. The VXLAN header is eight bytes long with three bytes or 24 bits as VNIs. So if you add up the VXLAN header of eight bytes with the outer UDP header and the outer IPv4 header and assuming no extensions are used, this adds 50 bytes of overhead for VXLAN if the VLAN tag is not used and 54 bytes if the VLAN tag is used. So in summary, this should be kept in mind when designing your data center network with overlay technologies. For overlay architectures also there is a consideration of which type of control plane you will use to advertise the endpoint reachability information. The first option is to use traditional flood and learn mechanism, but this will require multicast enabled underlay network. So in case you choose this option, VNIs are associated to multicast group addresses. This has to be configured on the switch side. When bump traffic is received, it is encapsulated in the overlay technology frame whose destination IP is the multicast group of the VNI. All VTEPs with VMs in a given VNI join the multicast group and receive bump traffic. You need to configure IGMP snooping and courier on the hypervisor phasing interfaces. This is required to optimize the delivery of layer two multicast traffic. PIM should also be enabled to ensure multicast traffic is also delivered to VTEPs in a different subnet from the source VTEP. One of the options is to use, the recommended option is to use PIM binder since all the VTEPs, they communicate with each other. It's a two-way communication. Each VNI should be assigned a multicast group and a rendezvous point in the spine nodes should be used for load balancing. So in this mode, the VTEP MAC or ARP tables are not maintained since their responsibility is given to the underlay. So this replication mechanism is similar to that tandem replication. The second option is to use an intelligent control plane protocol such as a VPN. And this is the second option. The third option is to use manual configuration where each VLAN is manually mapped to VNI. And the VTEPs are created manually as well. The last option is to use software-defined controllers such as SDN controllers such as VMware NSX or Nuage which use OVSDB or GENE for management and configuration of the hardware and software VTEPs. ALE on the switch 6900, it supports VXLAN and programming protocols and interfaces to be deployed with major SDN solutions. ALE is one of the few networking vendors to support SPV which is a standard Ethernet protocol. And the SPV fabric switch such as the 6900 will be able to run VXLAN in order to be integrated with an SDN solution and offer the gateway functionalities to connect legacy servers to virtual domains. Since our switches, they support OVSDB. As you can see here, we support any hypervisor and virtualization solution, OpenStack, Microsoft VMware, which will use an SDN controller with standard interface protocol to the data plane switches. Our 6900 can be deployed with SDN solutions with either VXLAN over a spinal leaf fabric with either link aggregation or SPV for the data plane. Our customers can also use our virtual chassis features to overcome the limitations of MCLAG. We will discuss overlay architectures and designs in another upcoming video, so stay tuned. So let us close with some key takeaways. When designing your data center, plan for the future by designing a scalable architecture. Use a top-down design approach based on the application requirements by calculating an acceptable over-subscription ratio and capacity requirements. Also, avoid adding complexity in your design unless it is necessary. This will make the network easier to troubleshoot and manage. And stretching subnets should not be considered unless absolutely required. Lastly, considerations should be taken when designing our underlying network, depending on which overlay technology should be used. Overlay architecture such as EVPN, which uses BGP extensions to transmit layer 2 or layer 3 reachability information, should be considered for medium to large deployments. Of course, there are a lot more concepts to cover in data center design, but I hope this session gave you a brief overview of the different concepts and technologies. Make sure to check out the data center reference design solution guide, which is available on Space Walker's resources database. And stay tuned for our upcoming session on EVPN and how this technology is used widely in data center. [00:46:05] Speaker ?: Thank you.