Internet-Draft EVPN and IPVPN Interworking April 2024
Rabadan, et al. Expires 13 October 2024 [Page]
Workgroup:
BESS Workgroup
Internet-Draft:
draft-ietf-bess-evpn-ipvpn-interworking-10
Published:
Intended Status:
Standards Track
Expires:
Authors:
J. Rabadan, Ed.
Nokia
A. Sajassi, Ed.
Cisco
E. Rosen
Individual
J. Drake
Independent
W. Lin
Juniper
J. Uttaro
AT&T
A. Simpson
Nokia

EVPN Interworking with IPVPN

Abstract

Ethernet Virtual Private Network (EVPN) is used as a unified control plane for tenant network intra and inter-subnet forwarding. When a tenant network spans not only EVPN domains but also domains where BGP VPN-IP or IP families provide inter-subnet forwarding, there is a need to specify the interworking aspects between BGP domains of type EVPN, VPN-IP and IP, so that the end to end tenant connectivity can be accomplished. This document specifies how EVPN interworks with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families for inter-subnet forwarding. The document also addresses the interconnect of EVPN domains for Inter-Subnet Forwarding routes. In addition, this specification defines a new BGP Path Attribute called D-PATH (Domain PATH) that protects gateways against control plane loops. D-PATH modifies the BGP best path selection for multiprotocol BGP routes of SAFI 128 and EVPN IP Prefix routes, and therefore this document updates the BGP best path selection in [RFC4271], but only for IPVPN and EVPN families.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 13 October 2024.

Table of Contents

1. Introduction and Problem Statement

EVPN is used as a unified control plane for tenant network intra and inter-subnet forwarding. When a tenant network spans not only EVPN domains but also domains where BGP VPN-IP or IP families provide inter-subnet forwarding, there is a need to specify the interworking aspects between the different families, so that the end to end tenant connectivity can be accomplished. This document specifies how EVPN should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families for inter-subnet forwarding. The document also addresses the interconnect of an EVPN domain to another EVPN domain for Inter-Subnet Forwarding routes. In addition, this specification defines a new BGP Path Attribute called D-PATH (Domain PATH) that protects gateways against control plane loops. Loops are created when two (or more) redundant gateway PEs interconnect two domains and exchange inter-subnet forwarding routes. For instance, if PE1 and PE2 are redundant gateway PEs interconnecting an IPVPN and an EVPN domain, gateway PE1 receives a VPN-IP route to prefix P and propagates the route into an EVPN IP Prefix to P. If gateway PE2 receives the EVPN IP Prefix route, it cannot propagate the route back to the IPVPN domain, or it would create a loop for prefix P.

D-PATH modifies the BGP best path selection for multiprotocol BGP routes of SAFI 128 and EVPN IP Prefix routes, and therefore this document updates the BGP best path selection procedures in [RFC4271] for IPVPN and EVPN families.

EVPN supports the advertisement of IPv4 or IPv6 prefixes in two different route types:

When interworking with other BGP address families (AFIs/SAFIs) for inter-subnet forwarding, the IP prefixes in those two EVPN route types must be propagated to other domains using different SAFIs. Some aspects of that propagation must be clarified. Examples of these aspects or procedures across BGP families are: route selection, loop prevention or BGP Path attribute propagation. The Interworking PE concepts are defined in Section 3, and the rest of the document describes the interaction between Interworking PEs and other PEs for end-to-end inter-subnet forwarding.

2. Conventions used in this document

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. Terminology and Interworking PE Components

This section summarizes the terminology related to the "Interworking PE" concept that will be used throughout the rest of the document.

   +-------------------------------------------------------------+
   |                                                             |
   |              +------------------+           Interworking PE |
   | Attachment   | +------------------+                         |
   | Circuit(AC1) | |  +----------+    |                MPLS/NVO tnl
 ----------------------*Bridge    |    |                    +------
   |              | |  |Table(BT1)|    |    +-----------+  / \     \
MPLS/NVO tnl +-------->|          *---------*           |<--> | Eth |
  -------+   |    | |  |Eth-Tag x +    |IRB1|           |  \ /     /
 / Eth  / \<-+    | |  +----------+    |    |           |   +------
|      |   |      | |     ...          |    |  IP-VRF1  |        |
 \      \ /<-+    | |  +----------+    |    |  RD2/RT2  |MPLS/NVO tnl
  -------+   |    | |  |Bridge    |    |    |           |   +------
   |         +-------->|Table(BT2)|    |IRB2|           |  / \     \
   |              | |  |          *---------*           |<--> | IP  |
 ----------------------*Eth-Tag y |    |    +-----*-----+  \ /     /
   |  AC2         | |  +----------+    |       AC3|         +------
   |              | |    MAC-VRF1      |          |              |
   |              +-+    RD1/RT1       |          |              |
   |                +------------------+          |  SAFIs       |
   |                                              |  1     +---+ |
 -------------------------------------------------+  128   |BGP| |
   |                                                 EVPN  +---+ |
   |                                                             |
   +-------------------------------------------------------------+
Figure 1: EVPN-IPVPN Interworking PE
                        GW1------------GW3
                      +------+       +------+
        +-------------|IP-VRF|       |IP-VRF|-------------+
       PE1            +------+       +------+            PE2
     +------+   DC1      |     WAN      |     DC2     +------+
 TS1-|IP-VRF|   EVPN     |    IPVPN     |     EVPN    |IP-VRF|-TS2
     +------+           GW2            GW4            +---+--+
        |             +------+       +------+             |
        +-------------|IP-VRF|       |IP-VRF|-------------+
                      +------+       +------+
                         +--------------+
            DOMAIN 1         DOMAIN 2       DOMAIN 3
        <---------------> <------------> <---------------->
Figure 2: Multiple domain DCI example
                             EVPN
        <------------------------------------------------->
                             BGP-LU
        <------------------------------------------------->

                       ASBR------------ASBR
                      +------+       +------+
        +-------------|      |       |      |-------------+
       PE1            +------+       +--+---+            PE2
     +------+   DC1      |     WAN      |     DC2     +------+
 TS1-|IP-VRF|   EVPN     |              |     EVPN    |IP-VRF|-TS2
     +------+          ASBR            ASBR           +---+--+
        |             +------+       +------+             |
        +-------------|      |       |      |-------------+
                      +------+       +------+
                         +--------------+

        <--------------------DOMAIN-1--------------------->
Figure 3: Single domain DCI example
                                +---+
                                |PE2|
                                +---+
                                 ^
            Interworking         |EVPN
                    PE    EVPN   v
                   +---+  IPVPN +--+       +---+
                   |PE1| <----> |RR| <---> |PE3|
                   +---+        +--+ IPVPN +---+
                 Composite
Figure 4: Interworking composite PE example
                            Interworking PE
                    +---+ EVPN   +---+ IPVPN  +---+
                    |PE2| <----> |PE1| <----> |PE3|
                    +---+        +---+        +---+
                                Gateway
Figure 5: Interworking gateway PE example

4. Domain Path Attribute (D-PATH)

The BGP Domain Path (D-PATH) attribute is an optional and transitive BGP path attribute.

Similar to AS_PATH, D-PATH is composed of a sequence of Domain segments. Each Domain segment is comprised of <domain segment length, domain segment value>, where the domain segment value is a sequence of one or more Domains, as illustrated in Figure 6. Each domain is represented by <DOMAIN-ID:ISF_SAFI_TYPE>.

Octets
0               1                    8                         n
+---------------+----------------//--+----//-------------------+
|Domain Segment |   Last Domain      |        Domain of Origin |
|    Length     |                    |                         |
+---------------+----------------//--+----//-------------------+
                 \__________________/
                             |
           Octets            v
           0                         6                7
           +------------------//-----+----------------+
           |    DOMAIN-ID            | ISF_SAFI_TYPE  |
           +------------------//-----+----------------+
           \________________________/
                        |
        Octets          v
        0     1     2     3     4     5     6
        +-----------------------+-----------+
        |        Global         |  Local    |
        |        Admin          |  Admin    |
        +-----------------------+-----------+
Figure 6: D-PATH Domain Segment
Table 1
Value Type
0 Gateway PE local ISF route
70 EVPN
128 SAFI 128

The BGP D-PATH attribute is supported on ISF routes of type IPVPN and EVPN and MUST NOT be advertised along with routes different from IPVPN and EVPN routes. By default, the BGP D-PATH attribute is not advertised and MUST be explicitly enabled by configuration on the Gateway PEs. In addition, D-PATH:

  1. Identifies the sequence of domains, each identified by a <DOMAIN-ID:ISF_SAFI_TYPE> through which a given ISF route of type IPVPN or EVPN has passed.

    • This attribute list MAY contain one or more segments.

    • The first entry in the list (leftmost) is the <DOMAIN-ID:ISF_SAFI_TYPE> from which a gateway PE is propagating an ISF IPVPN or EVPN route. The last entry in the list (rightmost) is the <DOMAIN-ID:ISF_SAFI_TYPE> from which a gateway PE received an ISF IPVPN or EVPN route without a D-PATH attribute (the Domain of Origin). Intermediate entries in the list are domains that the ISF IPVPN or EVPN route has transited.

    • As an example, an ISF IPVPN or EVPN route received with a D-PATH attribute containing a domain segment of {length=2, <6500:2:IPVPN>,<6500:1:EVPN>} indicates that the route was originated in EVPN domain 6500:1, and propagated into IPVPN domain 6500:2.

  2. It is added/modified by a gateway PE when propagating an update to a different domain (which runs the same or different ISF SAFI):

    • A gateway PE's IP-VRF, that connects two domains, belongs to two DOMAIN-IDs, e.g. 6500:1 for EVPN and 6500:2 for IPVPN.

    • Whenever a prefix arrives at a gateway PE in a particular ISF SAFI route, if the gateway PE needs to export that prefix to a BGP peer, the gateway PE MUST prepend a <DOMAIN-ID:ISF_SAFI_TYPE> to the list of domains in the received D-PATH, as long as the gateway PE works in Uniform-Propagation-Mode, as explained in Section 5.2 .

    • For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1 for EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P is received and P installed in the IP-VRF, the IPVPN route for P that is exported to an IPVPN peer will prepend the domain <6500:1:EVPN> to the previously received D-PATH attribute. Likewise, IP-VRF prefixes that are received from IP-VPN, will be exported to EVPN peers with the domain <6500:2:IPVPN> added to the segment.

    • In the above example, if the EVPN route is received without D-PATH, the gateway PE will add the D-PATH attribute with one segment {length=1, <6500:1:EVPN>} when re-advertising to domain 6500:2.

    • Within the Domain of Origin, the update does not contain a D-PATH attribute because the update has not passed through a gateway PE yet.

  3. For a local ISF route, i.e., a configured route or a route learned from a local attachment circuit, a gateway PE has three choices:

    1. It MAY advertise that ISF route without a D-PATH attribute into one or more of its configured domains, in which case the D-PATH attribute will be added by the other gateway PEs in each of those domains.

    2. It MAY advertise that ISF route with a D-PATH attribute into one or more of its configured domains, in which case the D-PATH attribute in each copy of the ISF route is initialized with an ISF_SAFI_TYPE of 0 and the DOMAIN-ID of the domain with which the ISF route is associated.

    3. It MAY advertise that ISF route with a D-PATH attribute that contains a configured domain specific to its local ISF routes into one or more of its configured domains, in which case the D-PATH attribute in each copy of the ISF route is initialized with a ISF_SAFI_TYPE of 0 and the DOMAIN-ID for the local ISF routes. This DOMAIN-ID MUST be globally unique and MAY be shared by two or more gateway PEs.

  4. An ISF IPVPN or EVPN route received by a gateway PE with a D-PATH attribute that contains one or more of its locally associated DOMAIN-IDs (for the IP-VRF) is considered to be a looped ISF route for the purpose of re-exporting the route to the adjacent domain in a Gateway PE. The ISF route in this case MUST be flagged as "looped", MUST NOT be exported, and MAY be installed in the IP-VRF only in case there is no better route after the best path selection (Section 6). The ISF_SAFI_TYPE is irrelevant for the purpose of loop detection of an ISF route. In other words, an ISF route is considered as a looped route if it contains a D-PATH attribute with at least one DOMAIN-ID matching a local DOMAIN-ID, irrespective of the ISF_SAFI_TYPE of the DOMAIN-ID.

    For instance, in the example of Figure 2, gateway GW1 receives TS1 prefix in two different ISF routes:

    • In an EVPN IP Prefix route with next-hop PE1 and no D-PATH attribute.

    • In a SAFI 128 route with next-hop GW2 and D-PATH = {length=1, <6500:1:EVPN>}, assuming that DOMAIN-ID for domain 1 is 6500:1.

    Gateway GW1 flags the SAFI 128 route as "looped" (since 6500:1 is a local DOMAIN-ID in GW1) and it will not install it in the tenant IP-VRF, since the route selection process selects the EVPN IP Prefix route due to a shorter D-PATH attribute (Section 6). Gateway GW1 identifies the route as "looped" even if the ISF_SAFI_TYPE value is unknown to GW1, i.e., any value different from the ones specified in this document).

  5. A DOMAIN-ID value on a gateway PE MAY be assigned for a peering domain or MAY be scoped for an individual tenant IP-VRF.

    • If allocated for a peering domain, the DOMAIN-ID applies to all tenant IP-VRFs for that domain.

    • If allocated for a specific tenant IP-VRF, the processing of the received D-PATH and its propagation is in the context of the IP-VRF DOMAIN-ID. Route leaking is a use-case where a per-IP-VRF DOMAIN-ID assignment is necessary. Suppose gateways PE1 and PE2 are attached to two different tenant IP-VRFs, IP-VRF-1 and IP-VRF-2. ISF SAFI routes advertised by gateway PE1 for IP-VRF-1 are received on gateway PE2 with DOMAIN-ID 6500:1. If the routes are leaked from IP-VRF-1 into IP-VRF-2 on PE2, and re-advertised back to PE1 in the context of IP-VRF-2, PE1 will not identify the route as a looped route. This is because PE1 processes the route in the context of IP-VRF-2, where DOMAIN-ID 6500:1 is not a local DOMAIN-ID.

  6. The number of domains in the D-PATH attribute indicates the number of gateway PEs that the ISF route update has transited. If one of the transit gateway PEs leaks a given ISF route between two local IP-VRFs, it MAY prepend a domain with a ISF_SAFI_TYPE of 0 for the leaked route when the route is exported into an ISF SAFI. In that case, the number of domains in the D-PATH attribute indicates the number of tenant IP-VRFs that the ISF route update has transited.

  7. The following error-handling rules apply to the D-PATH attribute:

    1. A received D-PATH attribute is considered malformed if it contains a malformed Domain Segment.

    2. A Domain Segment is considered malformed in any of the following cases:

      • The Domain Segment length of the last Domain Segment causes the D-PATH attribute length to be exceeded.

      • After the last successfully parsed Domain Segment there are less than eight octets remaining.

      • The Domain Segment has a Domain Segment Length of zero.

    3. A PE receiving an UPDATE message with a malformed D-PATH attribute SHALL apply "treat-as-withdraw" [RFC7606].

    4. Domains in the D-PATH attribute with unknown ISF_SAFI_TYPE values are accepted and not considered an error.

    5. In case a PE receives more than one D-PATH attribute with a route, the PE SHALL process the first one in the list and not store and propagate the others.

    6. D-PATH can be advertised with SAFI 128 and EVPN routes and MUST NOT be sent with any other AFI/SAFIs. If D-PATH is received along with routes of AFI/SAFI different from the IPVPN and EVPN families, the behavior treat-as-withdraw is applied [RFC7606].

  8. The use of D-PATH is restricted to "walled garden" Virtual Private Networks, and the operator MUST NOT turn on the generation of D-PATH along with IPVPN and/or EVPN routes if there are CEs attached to a PE (of any domain in the Virtual Private Network) that are connected to the Internet. In addition, a Gateway PE MUST support the removal of the D-PATH attribute on import and on export, based on configuration.

5. BGP Path Attribute Propagation across Domains

Based on its configuration, a gateway PE is required to propagate an ISF route between two domains that use the same or different ISF SAFI. This requires a definition of what a gateway PE has to do with BGP Path Attributes attached to the ISF route that the gateway PE is propagating. This section specifies the BGP Path Attribute propagation modes that a gateway PE may follow when receives an ISF route with ISF SAFI-x, installs the route in the IP-VRF and exports the ISF route into ISF SAFI-y. ISF SAFI-x and SAFI-y values MAY be the same values.

5.1. No-Propagation-Mode

This is the default mode of operation for gateway PEs that re-export ISF routes from a domain into another domain. In this mode, the gateway PE will simply re-initialize the BGP Path Attributes when propagating an ISF route, as it would for direct or local IP prefixes. This model may be enough in those use-cases where, e.g., the EVPN domain is considered an "abstracted" CE and remote IPVPN/IP PEs don't need to consider the original EVPN Attributes for path computations.

Since this mode of operation does not propagate the D-PATH attribute either, redundant gateway PEs are exposed to routing loops. Those loops may be resolved by policies and the use of other attributes, such as the Route Origin extended community [RFC4360], however not all the loop situations may be identified.

5.2. Uniform-Propagation-Mode

In this mode, the gateway PE simply keeps accumulating or mapping certain key commonly used BGP Path Attributes when propagating an ISF route. This mode is typically used in networks where EVPN and IPVPN SAFIs are used seamlessly to distribute IP prefixes.

The following rules MUST be observed by the gateway PE when propagating BGP Path Attributes:

  1. The gateway PE imports an ISF route in the IP-VRF and stores the original Path Attributes. The following set of Path Attributes SHOULD be propagated by the gateway PE when advertising the ISF route to a different domain (other BGP Path Attributes SHOULD NOT be propagated):

    • AS_PATH

    • D-PATH, only when advertising SAFI 128 and EVPN routes.

    • IBGP-only Path Attributes (when advertising to IBGP peers): LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID

    • MED

    • AIGP

    • Communities, Extended Communities, Large Communities and Wide Communities [I-D.ietf-idr-wide-bgp-communities], except in the exception cases detailed in point 4 of this section.

  2. When propagating an ISF route to a different IBGP peer, the gateway PE SHOULD keep the AS_PATH of the originating ISF route and add it to the destination ISF SAFI without any modification. When re-advertising to an EBGP peer, the gateway PE SHOULD keep the AS_PATH of the originating ISF route and prepend the IP-VRF's AS before sending the route.

  3. When propagating an ISF route to IBGP peers, the gateway PE SHOULD keep the IBGP-only Path Attributes from the originating route to the re-advertised route.

  4. As discussed in point 1, Communities, Extended Communities, Large Communities and Wide Communities SHOULD be preserved from the originating ISF route by the gateway PE. Exceptions of Extended Communities that SHOULD NOT be propagated are:

    1. BGP Encapsulation extended communities [RFC9012].

    2. Route Target extended communities. Route Targets are always initialized when readvertising an ISF route into a different domain, i.e., they are not propagated. The initialized Route Target in the re-advertised ISF route may or may not have the same value as the Route Target of the originating ISF route.

    3. All the extended communities of type EVPN.

    The gateway PE SHOULD NOT copy the above extended communities from the originating ISF route to the re-advertised ISF route.

  5. For a given ISF route, only the BGP Path Attributes of the best path can be propagated to another ISF route. If multiple paths are received for the same route in an ISF SAFI, the BGP best path selection will determine what the best path is, and therefore the set of Path Attributes to be propagated. Even if Equal Cost Multi-Path (ECMP) is enabled on the IP-VRF by policy, only the BGP Path Attributes of the selected best path are propagated.

5.3. Aggregation of Routes and Path Attribute Propagation

Instead of propagating a high number of (host) ISF routes between domains, a gateway PE that receives multiple ISF routes from a domain MAY choose to propagate a single ISF aggregate route into a different domain. In this document, aggregation is used to combine the characteristics of multiple ISF routes in such way that a single aggregate ISF route can be propagated to the destination domain. Aggregation of multiple ISF routes of one ISF SAFI into an aggregate ISF route is only done by a gateway PE.

Aggregation on gateway PEs may use either the No-Propagation-Mode or the Uniform-Propagation-Mode explained in Section 5.1 and Section 5.2, respectively.

When using Uniform-Propagation-Mode, Path Attributes of the same type code MAY be aggregated according to the following rules:

  • AS_PATH is aggregated based on the rules in [RFC4271]. The gateway PEs are not expected to receive AS_PATH attributes with path segments of type AS_SET [RFC6472]. Routes received with AS_PATH attributes including AS_SET path segments MUST NOT be aggregated.

  • An ISF aggregate route SHOULD NOT be advertised unless all the contributing ISF routes have the same D-PATH value. If there is at least one contributing ISF route that has different D-PATH, the gateway PE SHOULD advertise each contributing ISF route with its own D-PATH (prepended with the gateway's domain). An implementation MAY override this behavior, via policy, to advertise an ISF aggregate route without D-PATH in case the contributing routes did not have the same D-PATH value.

  • The Community, Extended Community and Large Community attributes of the aggregate ISF route MUST contain all the Communities/Extended Communities/Large Communities from all of the aggregated ISF routes, with the exceptions of the extended communities listed in Section 5.2 that are not propagated.

  • For other attributes, rules in [RFC4271] are followed.

Assuming the aggregation can be performed (the above rules are applied), the operator should consider aggregation to deal with scaled tenant networks where a significant number of host routes exists. For example, large Data Centers.

6. Route Selection Process for ISF Routes

A PE may receive an IP prefix in ISF routes with different ISF SAFIs, from the same or different BGP peer. It may also receive the same IP prefix (host route) in an EVPN MAC/IP Advertisement route and EVPN IP Prefix route. A route selection algorithm across all ISF SAFIs is needed so that:

For a given prefix advertised in one or more non-EVPN ISF routes, the BGP best path selection procedure will produce a set of "non-EVPN best paths". For a given prefix advertised in one or more EVPN ISF routes, the BGP best path selection procedure will produce a set of "EVPN best paths". To support EVPN/non-EVPN ISF interworking in the context of the same IP-VRF receiving non-EVPN and EVPN ISF routes for the same prefix, it is then necessary to run a tie-breaking selection algorithm on the union of these two sets. This tie-breaking algorithm begins by considering all EVPN and other ISF SAFI routes, equally preferable routes to the same destination, and then selects routes to be removed from consideration. The process terminates as soon as only one route remains in consideration.

The route selection algorithm must remove from consideration the routes following the rules and the order defined in [RFC4271], with the following exceptions and in the following order:

  1. Immediately after removing from consideration all routes that are not tied for having the highest Local Preference, any routes that do not have the shortest D-PATH are also removed from consideration. Routes with no D-PATH are considered to have a zero-length D-PATH. A BGP speaker MUST skip this rule for ISF SAFI routes that are not imported in an IP-VRF.

  2. Then regular [RFC4271] selection criteria is followed.

  3. At the end of the selection algorithm, if at least one route still under consideration is an EVPN MAC/IP Advertisement route, remove from consideration any EVPN IP Prefix routes.

  4. If Steps 1-3 leave Equal Cost Multi-Paths (ECMP) between non-EVPN and EVPN paths, the EVPN path MUST be considered (and the non-EVPN path removed from consideration). However, if ECMP across ISF SAFIs is enabled by policy, and one EVPN path and one non-EVPN path remain at the end of step 3, both path types MUST be used.

The above process modifies the [RFC4271] selection criteria for multiprotocol BGP routes with SAFI 128 and EVPN IP Prefix routes to include the shortest D-PATH so that operators minimize the number of Gateways and domains through which packets need to be routed. D-PATH does not modify the selection process for routes different from SAFI 128 or EVPN routes (received routes with other SAFIs get a treat-as-withdraw behavior as described in Section 4).

Example 1 - PE1 receives the following routes for IP1/32, that are candidate to be imported into IP-VRF-1:

{SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)}
{SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)}
{SAFI=128, Local-Pref=100, AS-Path=(100,200)}

Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200] (due to step 3, and no ECMP).

Example 2 - PE1 receives the following routes for IP2/24, that are candidate to be imported into IP-VRF-1:

{SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), MED=10}
{SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), MED=200}

Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS- Path=(100,200), MED=10} (due to step 1).

7. Composite PE Procedures

As described in Section 3, composite PEs are typically used in tenant networks where EVPN and IPVPN are both used to provide inter-subnet forwarding within the same composite domain.

Figure 7 depicts an example of a composite domain, where PE1/PE2/PE4 are composite PEs (they support EVPN and IPVPN ISF SAFIs on their peering to the Route Reflector), and PE3 is a regular IPVPN PE.

             +-----------------------------------+
             |                                   |
             |        MPLS                  IPVPN PE3
             |        Network              +----------+ IP3/24
             |                     IPVPN   |+------+  |   +---+
             |                      +----->||IP-VRF|------|CE3|
        Composite PE1               |      |+------+  |   +---+
       +---------------+            |      +----------+
       |      +------+ |  EVPN      v             |
       |      |IP-VRF| |  IPVPN   +--+            |
       | +----|      | | <------> |RR|            |
+---+  | |    +------+ |          +--+         Composite PE4
|CE2|----|MAC-VRF|     |          ^  ^         +---------+ IP4/24
+---+  | +-------+     |    EVPN  |  | EVPN    |+------+ |   +---+
       +---|-----------+    IPVPN |  | IPVPN   ||IP-VRF|-----|CE4|
           |  |              +----+  +-------->|+------+ |   +---+
    IP1/24 |  |              v                 +---------+
    +---+  |  |    +---------------+              |
    |CE1|--+  +----|      +------+ +--------------+
    +---+          |      |IP-VRF| |
      |            | +----|      | |
      |            | |    +------+ |
      +--------------|MAC-VRF|     |
                   | +-------+     |
                   +---------------+
                      Composite PE2
Figure 7: Composite PE example

In a composite domain with composite and regular PEs:

  1. The composite PEs MUST advertise the same IP prefixes in each ISF SAFI to the Route Reflector (RR). For example, in Figure 7, the prefix IP1/24 is advertised by PE1 and PE2 to the Route Reflector in two separate NLRIs, one for AFI/SAFI 1/128 and another one for EVPN.

  2. As an informative note, the Route Reflector does not forward EVPN routes to neighbors on which the EVPN SAFI is not enabled, and similarly, the Route Reflector does not forward IPVPN routes to neighbors on which the IPVPN SAFI is not enabled. For example, the Route Reflector does not forward EVPN routes to PE3 (since the Route Reflector does not have the EVPN SAFI enabled on its BGP session to PE3), whereas the IPVPN routes are forwarded to all the PEs.

  3. IPVPN PEs process and import IPVPN routes, as in [RFC4364]. As an example, PE3 receives only the IPVPN route for IP1/24 and resolves the BGP next-hop to an MPLS tunnel (with IP payload) to PE1 and/or PE2.

  4. Composite PEs MUST process routes for the same prefix coming from different ISF SAFI routes, and perform route selection.

    • As an example, PE4 receives IP1/24 encoded in EVPN and another ISF SAFI route (EVPN IP Prefix route and IPVPN). The route selection follows the procedures in Section 6.

    • Assuming an EVPN route is selected, PE4 resolves the BGP next-hop to an MPLS tunnel (with Ethernet or IP payload) to PE1 and/or PE2. As described in Section 3, two EVPN PEs may use tunnels with Ethernet or IP payloads to connect their IP-VRFs, depending on the [RFC9136] model implemented.

    • The other composite PEs (PE1 and PE2) receive also the same IP prefix via EVPN and IPVPN SAFIs and they also follow the route selection in Section 6.

  5. When a given route has been selected as the route for a particular packet, the transmission of the packet MUST be done according to the rules for that route's AFI/SAFI.

  6. As an informative note, in composite domains, such as the one in Figure 7, the EVPN advanced forwarding features will only be available to composite and EVPN PEs (assuming they select an EVPN IP Prefix route to forward packets for a given IP prefix), and not to IPVPN PEs. For example, assuming PE1 sends IP1/24 in an EVPN and an IPVPN route and the EVPN route is the best one in the selection, the recursive resolution of the EVPN IP Prefix routes can only be used in PE2 and PE4 (composite PEs), and not in PE3 (IPVPN PE). As a consequence of this, the indirection provided by the EVPN IP Prefix route recursive resolution and its benefits in a scaled network, will not be available in all the PEs in the network.

8. Gateway PE Procedures

Section 3 defines a gateway PE as an Interworking PE that is attached to two (or more) domains and propagates ISF routes between those domains. Examples of gateway PEs are Data Center gateways connecting domains that make use of EVPN and other ISF SAFIs for a given tenant. The gateway PE procedures in this document provide an interconnect solution for ISF routes and complement the gateway definition of [RFC9014], which focuses on the interconnect solution for Layer 2. This section applies to the interconnect of two domains that use different ISF SAFIs (e.g., EVPN to IPVPN), as well as the interconnect of two domains of the same ISF SAFI (e.g., EVPN to EVPN). Figure 8 illustrates a gateway PE use-case, in which PE1 and PE2 (and PE3/PE4) are gateway PEs interconnecting domains for the same tenant.

   <----EVPN---->    <----------IPVPN--------->   <----EVPN---->
     6500:1:EVPN             6500:2:IPVPN           6500:3:EVPN
<DOMAIN-ID:ISF_SAFI_TYPE>
                      +-----------------------+
               Gateway PE1              Gateway PE3
               +----------+             +----------+
   +-----------|+------+  |  MPLS tnls  |+------+  |------------+
   |           ||IP-VRF|  |             ||IP-VRF|  |            |
 PE5           |+------+  |             |+------+  |           PE6
+------+       +----------+             +----------+         +------+
|IP-VRF| NVO tnls |   |                       |  | NVO tnls  |IP-VRF|
|      |          |   |                       |  |           |      |
+------+       +----------+             +----------+         +------+
IP1/24-->      |+------+  |             |+------+  |            |
   |           ||IP-VRF|  |             ||IP-VRF|  |            |
   +-----------|+------+  |             |+------+  |------------+
               +----------+             +----------+
               Gateway PE2   +------+   Gateway PE4
                     +-------|IP-VRF|---------+
                             |      |
                             +------+
                               PE7
Figure 8: Gateway PE example

The procedures for a gateway PE enabled for ISF SAFI-x and ISF SAFI-y on the same IP-VRF follow:

  1. A gateway PE that imports an ISF SAFI-x route to prefix P in an IP-VRF, MUST export P in ISF SAFI-y if:

    1. P is installed in the IP-VRF - which means the SAFI-x route is well-formed, valid and the best one for P - and

    2. PE has a BGP peer for SAFI-y (enabled for the same IP-VRF) and

    3. The advertisement is allowed by policy and

    4. ISF SAFI-x and ISAF SAFI-y are any of the types defined in Section 3. Note that SAFI-x and SAFI-y MAY have the same value.

    In the example of Figure 8, gateway PE1 and PE2 receive an EVPN IP Prefix route with IP1/24, install the prefix in the IP-VRF and re-advertise it using SAFI 128.

  2. A gateway PE that receives an ISF SAFI-x route to prefix P in an IP-VRF MUST NOT export P in ISF SAFI-y if:

    1. The ISF SAFI-x route is not well-formed or valid. Rules to determine if a route is well-formed or valid for a given ISF SAFI are defined by the specification of each ISF SAFI. As an example, an EVPN IP Prefix route received with non-zero ESI and GW IP values, at the same time, is not valid as per [RFC9136], section 3.2.

    2. The ISF SAFI-x route contains a D-PATH attribute with one or more of the gateway PE's locally associated domains for the IP-VRF. In this case the route is considered to be a looped ISF route, as described in Section 4 and hence MUST NOT be exported in ISF SAFI-y.

Once the gateway PE determines that P must be exported, P will be advertised using ISF SAFI-y as follows:

  1. If Uniform-Propagation-Mode is enabled Section 5.2, the D-PATH attribute MUST be included if SAFI-y is equal to 128 or EVPN, so that loops can be detected in remote gateway PEs. When a gateway PE propagates an ISF route between domains, it MUST prepend a <DOMAIN-ID:ISF_SAFI_TYPE> to the received D-PATH attribute. The DOMAIN-ID and ISF_SAFI_TYPE fields refer to the domain over which the gateway PE received the IP prefix and the ISF SAFI of the route, respectively. If the received IP prefix route did not include any D-PATH attribute, the gateway IP MUST add the D-PATH when readvertising. The D-PATH in this case will have only one segment on the list, the <DOMAIN-ID:ISF_SAFI_TYPE> of the received route.

    In the example of Figure 8, gateway PE1/PE2 receive the EVPN IP Prefix route with no D-PATH attribute since the route is originated at PE5. Therefore PE1 and PE2 will add the D-PATH attribute including <DOMAIN-ID:ISF_SAFI_TYPE> = <6500:1:EVPN>. Gateways PE3/PE4 will propagate the route again, now prepending their <DOMAIN-ID:ISF_SAFI_TYPE> = <6500:2:IPVPN>. PE6 receives the EVPN IP Prefix routes with D-PATH = {<6500:2:IPVPN>,<6500:1:EVPN>} and can use that information to make BGP path decisions.

  2. The gateway PE MAY use the Route Distinguisher of the IP-VRF to readvertise P in the ISF SAFI-y.

  3. The label allocation used by each gateway PE is a local implementation matter. The IP-VRF advertising IP prefixes for EVPN and another ISF SAFI may use a label per-VRF, per-prefix, etc.

  4. The gateway PE MUST be able to use the same or different set of Route Targets per domain on the same IP-VRF. In particular, if different domains use different set of Route Targets for the same tenant, the gateway PE MUST be able to import and export routes with the different sets.

  5. Even though Figure 8 only shows two domains per gateway PE, the gateway PEs may be connected to more than two domains.

  6. There is no limitation of gateway PEs that a given IP prefix P can pass through until it reaches a given PE.

  7. If the gateway PE uses Uniform-Propagation-Mode for BGP Path Attribute propagation, besides the processing of D-PATH described in point "a", the rules in Section 5.2 are followed.

  8. As an informative note, if P was originated in an EVPN domain but traversed a different ISF SAFI domain (or domains), it will lose EVPN-specific attributes that are used in advanced EVPN procedures. For example, even if PE1 advertises IP1/24 along with a given non-zero ESI (for recursive resolution to that ESI), when PE6 receives the IP prefix in an EVPN route, the ESI value will be zero. This is because the route traverses an ISF SAFI domain that is different from EVPN.

9. Interworking Use-Cases

While Interworking PE networks may well be similar to the examples described in Section 7 and Section 8, in some cases a combination of both functions may be required. Figure 9 illustrates an example where the gateway PEs are also composite PEs, since not only they need to propagate ISF routes between domains (from EVPN SAFI to IPVPN and/or EVPN SAFIs), but they also need to interwork with IPVPN-only PEs in a domain with a mix of composite and IPVPN-only PEs.

                      +-----------------------------------+
                      |                                   |
                      |        MPLS                 IPVPN PE3
                      |        Network              +---------+
                      |                     IPVPN   |+------+ |
                      |                      +----->||IP-VRF|---TS3
               (GW+composite) PE1            |      |+------+ |
                +---------------+            |      +---------+
                |      +------+ |  EVPN      v            |
                |      |IP+VRF| |  IPVPN  +--+            |
                | +----|      | | <------>|RR|            |
       +--------| |    +------+ |         +--+       Composite PE4
       |        | |MAC+VRF|     |         ^  ^        +---------+
       |        | +-------+     |   EVPN  |  | EVPN   |+------+ |
    +----+      +---------------+   IPVPN |  | IPVPN  ||IP-VRF|---TS4
TS1-|NVE1|             |             +----+  +------->|+------+ |
    +----+             |             v                +---------+
       |    EVPN DC    |    +---------------+             |
       |    NVO tnls   +----|      +------+ |-------------+
       |                    |      |IP+VRF| |
       |                    | +----|      | |
       |                    | |    +------+ |
       |     +----+         | |MAC+VRF|     |
       +-----|NVE2|---------| +-------+     |
             +----+         +---------------+
               |           (GW+composite) PE2
              TS2
Figure 9: Gateway and composite combined functions - example

In the example above, PE1 and PE2 MUST follow the procedures described in Section 7 and Section 8. Compared to the example in Section 8, PE1 and PE2 now need to also propagate ISF routes from EVPN to EVPN, in addition to propagating prefixes from EVPN to IPVPN.

It is worth noting that PE1 and PE2 will receive TS4's IP prefix via IPVPN and EVPN IP Prefix routes. When readvertising to NVE1 and NVE2, PE1 and PE2 will consider the D-PATH rules and attributes of the selected route for TS4 (Section 6 describes the Route Selection Process).

10. BGP Error Handling on Interworking PEs

An Interworking PE (acting as gateway PE or composite PE) observes the following error-handling procedures for ISF routes:

If a gateway PE is set to propagate BGP Path Attributes for ISF routes across domains, the procedures in Section 5.2 guarantee that a BGP speaker does not receive UPDATES with well-formed but unexpected BGP Path Attributes. If a gateway PE fails to follow the propagation rules in Section 5.2 and propagates some BGP Path Attributes erroneously, the receiving PEs follow the specifications for the specific ISF route type and BGP Path Attribute. Some (but not all) examples follow:

11. Conclusion

This document describes the procedures required in PEs that process and advertise ISF routes for a given tenant. In particular, this document defines:

The above procedures provide an operator with the required tools to build large tenant networks that may span multiple domains, use different ISF SAFIs to handle IP prefixes, in a deterministic way and with routing loop protection.

12. Security Considerations

In general, the security considerations described in [RFC9136] and [RFC4364] apply to this document.

Section 4 introduces the use of the D-PATH attribute, which provides a security tool against control plane loops that may be introduced by the use of gateway PEs that propagate ISF IPVPN/EVPN routes between domains. A correct use of the D-PATH will prevent control plane and data plane loops in the network, however an incorrect configuration of the DOMAIN-IDs or an inconsistent support of D-PATH on the Gateway PEs may lead to the detection of false route loops, the blackholing of the traffic or may result in inconsistent and sub-optimal routing. An attacker may benefit of this transitive attribute to propagate the wrong domain information across multiple domains.

Section 4 restricts the use of D-PATH to IPVPN and EVPN routes in "walled garden" Virtual Private Networks. An upgraded PE removes D-PATH from the BGP Path Attributes before advertising an IP Prefix to a CE in a SAFI 1 route. However, if D-PATH is received by a non-upgraded IPVPN PE that has an attached CE connected to the Internet, the PE may incorrectly propagate the D-PATH attribute in a SAFI 1 route to the CE, and the D-PATH attribute may then escape out of the "walled garden" to the Internet. This may happen when the IPVPN PE re-exports a route directly, or via route leaking between IP-VRFs. Since D-PATH is a transitive attribute, if not upgraded to understand D-PATH, the CE may propagate the attribute to the Internet. However, since the attribute does not change the best path selection for SAFI 1 routes, D-PATH cannot create loops or inconsistent routing in the Internet. Upgraded Internet routers receiving the D-PATH attribute in a SAFI 1 route will apply the treat-as-withdraw behavior, as discussed in Section 4. As an additional security mechanism, a PE following this specification that receives an ISF EVPN or IPVPN route from a non-upgraded PE should discard the route via policy if the route contains the D-PATH attribute.

In addition, Section 5.2 introduces the propagation of BGP Path Attributes between domains on gateway PEs. Without this mode of propagation, BGP Path Attributes are re-initialized when re-exporting ISF routes into a different domain, and the operator does not have the end-to-end visibility of a given ISF route path. However, the Uniform Propagation mode introduces the capability of propagating BGP Path Attributes beyond the ISF SAFI scope. While this is a useful tool to provide end-to-end visibility across multiple domains, it can also be used by an attacker to propagate wrong (although correctly formed) BGP Path Attributes that can influence the BGP path selection in remote domains. An implementation can also choose Section 5.1 (No-propagation mode) to minimize the risks derived from propagating incorrect attributes, but again, this mode of operation will prevent the receiver PE from seeing the attributes that the originator of the route intended to convey in the first place.

13. IANA Considerations

This document defines a new BGP path attribute known as the BGP Domain Path (D-PATH) attribute.

IANA has assigned a new attribute code type from the "BGP Path Attributes" subregistry under the "Border Gateway Protocol (BGP) Parameters" registry:

Path Attribute Value    Code                       Reference
--------------------    ------------------------   ---------------
36                      BGP Domain Path (D-PATH)   [This document]

14. Acknowledgments

The authors want to thank Russell Kelly, Dhananjaya Rao, Suresh Basavarajappa, Mallika Gautam, Senthil Sathappan, Arul Mohan Jovel, Naveen Tubugere, Mathanraj Petchimuthu, Eduard Vasilenko, Amit Kumar, Mohit Kumar and Lukas Krattiger for their review and suggestions. Thanks to Sue Hares and Jeff Haas as well, for their detailed review to clarify the procedures of the D-PATH attribute.

15. Contributors

16. References

16.1. Normative References

[RFC7432]
Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, , <https://www.rfc-editor.org/info/rfc7432>.
[RFC8365]
Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., Uttaro, J., and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, , <https://www.rfc-editor.org/info/rfc8365>.
[RFC4271]
Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, , <https://www.rfc-editor.org/info/rfc4271>.
[RFC4364]
Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, , <https://www.rfc-editor.org/info/rfc4364>.
[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.
[RFC7606]
Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, DOI 10.17487/RFC7606, , <https://www.rfc-editor.org/info/rfc7606>.
[RFC4760]
Bates, T., Chandra, R., Katz, D., and Y. Rekhter, "Multiprotocol Extensions for BGP-4", RFC 4760, DOI 10.17487/RFC4760, , <https://www.rfc-editor.org/info/rfc4760>.
[RFC9136]
Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and A. Sajassi, "IP Prefix Advertisement in Ethernet VPN (EVPN)", RFC 9136, DOI 10.17487/RFC9136, , <https://www.rfc-editor.org/info/rfc9136>.
[RFC9135]
Sajassi, A., Salam, S., Thoria, S., Drake, J., and J. Rabadan, "Integrated Routing and Bridging in Ethernet VPN (EVPN)", RFC 9135, DOI 10.17487/RFC9135, , <https://www.rfc-editor.org/info/rfc9135>.
[RFC9252]
Dawra, G., Ed., Talaulikar, K., Ed., Raszuk, R., Decraene, B., Zhuang, S., and J. Rabadan, "BGP Overlay Services Based on Segment Routing over IPv6 (SRv6)", RFC 9252, DOI 10.17487/RFC9252, , <https://www.rfc-editor.org/info/rfc9252>.

16.2. Informative References

[RFC4360]
Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, , <https://www.rfc-editor.org/info/rfc4360>.
[RFC9012]
Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, , <https://www.rfc-editor.org/info/rfc9012>.
[RFC6472]
Kumari, W. and K. Sriram, "Recommendation for Not Using AS_SET and AS_CONFED_SET in BGP", BCP 172, RFC 6472, DOI 10.17487/RFC6472, , <https://www.rfc-editor.org/info/rfc6472>.
[RFC4659]
De Clercq, J., Ooms, D., Carugi, M., and F. Le Faucheur, "BGP-MPLS IP Virtual Private Network (VPN) Extension for IPv6 VPN", RFC 4659, DOI 10.17487/RFC4659, , <https://www.rfc-editor.org/info/rfc4659>.
[RFC8950]
Litkowski, S., Agrawal, S., Ananthamurthy, K., and K. Patel, "Advertising IPv4 Network Layer Reachability Information (NLRI) with an IPv6 Next Hop", RFC 8950, DOI 10.17487/RFC8950, , <https://www.rfc-editor.org/info/rfc8950>.
[RFC9014]
Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi, A., and J. Drake, "Interconnect Solution for Ethernet VPN (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014, , <https://www.rfc-editor.org/info/rfc9014>.
[I-D.ietf-idr-wide-bgp-communities]
Raszuk, R., Haas, J., Lange, A., Decraene, B., Amante, S., and P. Jakma, "BGP Community Container Attribute", Work in Progress, Internet-Draft, draft-ietf-idr-wide-bgp-communities-11, , <https://datatracker.ietf.org/doc/html/draft-ietf-idr-wide-bgp-communities-11>.

Authors' Addresses

J. Rabadan (editor)
Nokia
520 Almanor Avenue
Sunnyvale, CA 94085
United States of America
A. Sajassi (editor)
Cisco
225 West Tasman Drive
San Jose, CA 95134
United States of America
E. Rosen
Individual
J. Drake
Independent
W. Lin
Juniper
J. Uttaro
AT&T
A. Simpson
Nokia