Using BGP to announce cloud-native workloads

To realize the goal of dynamically obtaining and announcing layer-3 network identity (IPv4 addresses) for cloud-native workloads, we propose an approach that utilizes the BGP routing protocol.

In a typical network design, these two steps are necessary:

1. Add an EIGRP “network” statement at the access-layer, for the wider "service" subnet (eg. 192.168.0.0/24). Note that "access-to-distribution" dynamic routing is achieved using EIGRP in our datacenter.

2. Considering that a cloud-native load-balancer such as MetalLB announces the /32 service identity with next-hop value of the primary interface, there is no need add a network statement.  Instead, a secondary address/mask is specified at default-gateway for the access VLAN's layer-3 interface spec to allow this gateway to send ARP requests on the access network upon receiving traffic destined to the service subnet.

eBGP multi-hop between MetalLB and “distribution” switch will receive /32 with the new AS_PATH, and extended-community attributes (optional).  The distribution switch can now resolve the next hop.  Note that there is no need to establish a BGP relationship with “access” layer.  Additionally, we must setup the "service subnet" as a "secondary" IP address/mask specification on the gateway's layer-3 interface. We typically employ a pair of switches with HSRP enabled for high availability.

This method requires us to aggregate Linux BGP speakers with MetalLB at rack or server-room level. This represents a significant new capability for publishing services from cloud-native environments such as Kubernetes.

Typically, when a router receives an eBGP route where the next-hop is unknown, the route is not installed in the table, or propagated.

NEXT_HOP is inaccessible

We solved this problem by using an EIGRP network statement for the service prefix, one that be obtaining using APIs from a Network Identity Manager such as Infoblox.

If the router doesn't know how to reach a route's next hop, a recursive lookup will fail, and the route can't be added to BGP. For example, if a BGP router receives a route for 192.168.0.11/32 with a NEXT_HOP attribute of 192.168.0.1, but doesn't have an entry in its routing table for a subnet containing 192.168.0.1, the received route for 192.168.0.11 is useless and won't be installed in the routing table.

If however, MetalLB announces the prefix with the next-hop value of the primary interface (10.205.32.15 in this example), then the IGP (EIGRP) between "access" and "distribution" layers will ensure that next-hop is reachable, and the route is successfully installed.

BGP configuration example at the network layer router:

    router bgp 65001
    neighbor 10.205.32.15 remote-as 65002

    address-family ipv4 unicast

      neighbor 10.205.32.15 ebgp-multihop
      neighbor 10.205.32.15 activate
      neighbor 10.205.32.15 prefix-list service-subnets in

Prefix-list
`ip prefix-list service-subnets seq 100 permit 192.168.0.0/24 ge 32`

EIGRP configuration at the "access" layer switches

    router eigrp 100
      network 192.168.0.0/24

Layer-3 Gateway at "access" layer
    interface vlan 3032
      ip address 10.032.2/22
      ip address 192.168.0.1/24 secondary
        hsrp 32
          ip 10.0.32.1


MetalLB configuration

    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: metallb-system
      name: config
    data:
      config:
        peers:
        - peer-address: 10.0.0.1
          peer-asn: 65001
          my-asn: 65002
        address-pools:
        - name: default
          protocol: bgp
          addresses:
          - 192.168.0.0/24
          bgp-advertisements:
          - aggregation-length: 32
            localpref: 100
            communities:
            - no-advertise
          - aggregation-length: 24
        bgp-communities:
          no-advertise: 64512-64534

Fibre Channel switching demystified at a basic level

Fiber Channel has been compared to Ethernet and for good reasons. Both are OSI L2 protocols, and both use globaly unique physical addresses, but not in the same way! While MAC addresses are always present in ethernet traffic, FC WWNs are quite a bit different.  They never appear in the FC headers.

Let's understand the underlying FC switching fabric first to put things in perspective. When multiple FC switches are connected together using E_ports, they form a switch fabric. The domain ID is unique to each switch in the fabric and there lies the first problem with scalability: only 239 are available. The lowest [PS_priority+WWWN] becomes the PS (principal switch). The PS selection process occurs when the E_ports are first connected, and the BF (the mundane sounding "build fabric") frames are exchanged. If two switches contain the same domain ID then the link between the two is "isolated". The domain id can be chosen at random, unless it is administratively set. This is required to generate the initial discovery traffic such as BF, EFP, SW_ACC frames. If you are really curious, the S_id and D_id (FCIDs) in these frames are always to set to 0xFFFFFD which is the fabric controller address.

So, given the 239 domain IDs in any given fabric, how to we break through the limit to scale up? Enter NPV or N-port virtualization. An NPV enabled switch does not take up a domain ID, but instead, relays the FLOGIs comming received on F_ports (from N_ports on hosts) up to the core switch via NP_ports. In other words, ports on the NPV edge switch that connect to the F_ports on the core are always set to type "NPV". NPV mode and Fabric mode are mutually exclusive, and a reboot is necessary when selecting either mode.

What is NPIV? N_port id virtualization is a feature that allows F_ports to accept multiple FLOGI requests from the same N_port and assign FCIDs accordingly.  Ask: are NPIV and NPV mutually exclusive on a any given device? Hint: NPIV is a server feature, while NPV is a switch feature, typically. Whether it is NPV enabled switch or a NPIV enabled host, the F_port essentially behaves the same and accepts multiple FLOGIs. So, an NPV enabled switch looks exactly like an NPIV enabled host to a F_port. Phew! Enough already, right?

So, how does FLOGI work? What is the initial FCID in the frame that comes out of an N_port going toward the F_port? Ans: 0x000000 - this is the initial value of the FCID, and this frame is sent to 0xFFFFFE which represents the FLOGI server. The payload contains the WWNN and service parameters. The FLOGI servers assigns the FCID (N_port ID) and BB_Credit.

Next, the host can use its newly acquired N-port ID to continue with the PLOGI process where it sends its WWPN to FCID map. The destination FCID is that of the FCNS: 0xFFFFFC. The FCNS now registers this information it its database and exposes this to other devices according to zoning that has been configured.

Try the command: "show fcns database" if you SAN uses a Cisco switch.

Troubleshooting CRC and input errors with Nexus 5000 in transit path

When you encounter CRC (input) errors on an upstream switch interface, and you have a Nexus 5000 in the transit path (downstream), MTU Stomping by the latter is likely a factor.

On a Nexus 5000 switch, when a frame is received on a 10 Gb/s interface, it is considered to be in the cut-through path.  If there is no "network qos" type policy with jumbo frame support, and the hosts connected this switch generate jumbo frames, we will encounter the following situation:

There are logical and physical causes for the Nexus 5000 to drop a frame. There are also situations when a frame cannot be dropped because of the cut-through nature of the switch architecture. If a drop is necessary, but the frame is being switched in a cut-through path, then the only option is to stomp the Ethernet frame check sequence (FCS). Stomping a frame involves setting the FCS to a known value that does not pass a CRC check. This causes subsequent CRC checks to fail later in the path for this frame. A downstream store-and-forward device, or a host, will be able to drop this frame.

"show queueing interface ethernet x/y" will reveal the hardware MTU.  This setting is enforced by the application of a "network qos" type policy-map under "system qos" section of running configuration.

IP subnetting unraveled

Consider the following address in CIDR notation:

172.19.235.75 /13

1) What is the network address?

2) What is the broadcast address?

First, determine the subnet mask in dotted decimal notation: 255.248.0.0

Next, calculate: IP_Address AND Mask.  This reveals the network address as 172.16.0.0 (NET_address)

Then, determine the wildcard mask (WC_mask) by inverting 1's and 0's in the subnet mask: 0.7.255.255

Finally, calculate: NET_address OR WC_mask, and this reveals the broadcast address: 172.23.255.255

Now, practice this method on 10.0.0.100 /28 and if you are feeling adventurous, try 192.168.255.254 /22

Wireless AP 1142 - Convert to controller based or standalone mode

If you are faced with the task of installing 1100 series APs into a controller environment but somehow "standalone" APs were procured, don't panic! Here is the fix:

1. Note the AP's IP address by "show cdp neigh detail"
2. http:// to the address to manage (Cisco/Cisco... case sensitive ID and pass!)
3. Go to the software tab and update to the correct image

If your controller is a hop or two or more away, don't forget to adjust DHCP option 43 and option 60 values!

Cisco 2600 AP - Convert to Stand-Alone mode

Follow these simple steps:

1. Download the stand-alone image from Cisco.com (.tar)
2. Rename the file to ...tar.default
3. Make sure you have a console connection to the 2600 AP - this will you verify the name of the file the AP expects to boot from using TFTP.
4. Change your IP address to 10.0.0.2 /8
5. Run a TFTP  server on your laptop or PC
6. Plug the AP and your PC into two ports a switch that are in the same VLAN (no layer3 SVI necessary, I recommend creating a temporary VLAN such as 99 for example)
7. Make sure that the AP is connected to a PoE port if you don't have a power brick, BUT before you do that, press the MODE button and hold it down as the AP boots!!
8. Watch the console messages for the correct filename - ...tar.default should match exactly.
9. AP's LED will go blue and then after 15 seconds will go RED - you can let go of the MODE button now!
10. AP LED will blink green and it will intialize in stand-alone mode, and obtain a DHCP address.
11. You can manage it using http://

Cisco ASA: "LAND" attack - track false positves

A LAND attack is characterized by an IP packet whose source and destination IP addresses are the same. HOwever, sometimes on the ASA platform, it is possible to see false postives.


Consider the following example:

Jan 05 2012 16:21:36: %ASA-2-106017: Deny IP due to Land Attack from 192.168.2.71 to 192.168.2.71

Jan 05 2012 16:21:37: %ASA-2-106017: Deny IP due to Land Attack from 192.168.2.71 to 192.168.2.71

What is really happening?

Upon closer examination, you notice that you have static tranlation setup:

static (inside, outside) 192.168.2.71 172.16.2.71

Now let's capture traffic b/w the private address and translated IP, and lo and behold, the mystery is solved! The server inside is trying to access "itself via its public IP address" (perhaps via a script that is running).

3 packets captured

1: 16:20:37.032469 172.16.2.71.58126 > 192.168.2.71.80: S 1304500266:1304500266(0) win 5840

2: 16:21:36.938168 172.16.2.71.58128 > 192.168.2.71.80: S 4035860468:4035860468(0) win 5840

3: 16:21:37.173559 172.16.2.71.58129 > 192.168.2.71.80: S 4123968769:4123968769(0) win 5840

3 packets shown   Notice how the time-stamps match!

Load balancing across two ISPs with ASA

Ah! You are the security engineer at a medium sized bank, and perhaps you want to expand your Internet access to include a 2nd ISP. You are already paying $$ for the 10Mbps and now you are going to cough up even more $$$ for the new 25 Mbps service. Your websites and email servers are well set, and they are using static translations on the first ISP public address. How can you bring the 2nd ISP online, load-balance traffic across both links but preserve what you have built thus far?


You configure a select set of static routes (with SLA tracking if necessary) and this forces some outbound traffic off to the new link:

route outside-new 128.0.0.0 255.0.0.0 x.x.x.x

Just select a few more of your favorite Internet targets.  You can enable the SLA tracking feature to add an element of dynamism!

and so on...


But, you notice that connections for the outside are not completing. You setup "logging console notifications" and observe uRPF drops! Now that's easy to explain: traffic to your static translations will come in via the old circuit but new more specific routes break uRPF on the ASA. The solutions is to check and disable "ip verify reverse-path" and Voila! you are back in business!

ERSPAN with Nexus 1000V

The value of ERSPAN cannot be overstated!  It provides the much needed visibility into virtualized environments that are rapidly becoming the de facto server paradigm these days.

Encapsulated remote span work by tunnelling packets to a remote analyzer platform such as the Network Analysis Module (in this case its a NAM blade in a Catalyst 6500).

The key to picking up copies of traffic from inside the virtual world is the "l3-control" capability in a Nexus 1000V port-profile as seen here:

port-profile type vethernet esx_300
  capability l3control  vmware port-group
  switchport mode access
  switchport access vlan 300
  no shutdown
  system vlan 300
  state enabled
And notice that it appears in the "system-uplink" profile:

port-profile type ethernet system-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 300-304
  pinning control-vlan 0
  pinning packet-vlan 0
  mtu 9000
  channel-group auto mode on mac-pinning
  no shutdown
  system vlan 300,303
  state enabled
Next, we costruct the ERSPAN session as follows:

monitor session 64 type erspan-source
  description n1K_ERSPAN_example
  source vlan 300,304 both
  destination ip 172.31.255.254
  erspan-id 700
  ip ttl 64
  ip prec 0
  ip dscp 0
  mtu 1500
  header-type 2
  no shut
We selected an ERSPAN id of 700 which helps identify the session on the remote NAM platform.

Assuming that IP routing is active, the ERSPAN session will show up automatically on the NAM under "data sources".  You will be able to specify filters and analyze traffic from any VLAN that the Nexus 1000V controls within the virtual world.  Here we are using VLAN 304 as an example.

Private VLANs de-constructed in 50 words!

Private VLANs can be de-constructed into 3 types of ports

"P" ports

"C" ports

"I" ports

Promiscuous "P" ports - no restrictions

Community "C" ports can talk to other "C" ports in same community, and to "P" ports

Isolated "I" ports can only talk to "P" ports

Phone Proxy with ASA 8.2

ASA 8.2 Phone Proxy feature allows the security appliance to proxy Certificate Authority function on behalf of the CUCM.  This allows phones registering from the outside to contact the ASA which then proxies the connection to the CUCM.  The “media termination point” feature, along with the ‘Certificate Trust List’ CTL on the ASA ensures that media streams are encrypted (SRTP).

Due to the fact that the “Phone Proxy” ASA is separate from the “Internet ASA” at Securiosity Companies, “outside NAT” had to be implemented to ensure that Phone-to-CUCM” traffic was not asymmetrically routed.  Please review the configuration on the ASA (contained in this document) for specific details of this solution.

We have also modified a service policy to inspect SIP, and the policy is attached to the “outside” interface.

The only ACL entry relaxed here is TFTP traffic to the CUCM’s translated address on the “outside” address of the ASA.



Trustpoint for MIC (7941 and 7961 phones)

crypto ca trustpoint CAP-RTP-001_trustpoint

 enrollment terminal

 crl configure

crypto ca trustpoint CAP-RTP-002_trustpoint

 enrollment terminal

 crl configure

crypto ca trustpoint Cisco_Manufacturing_CA_trustpoint

 enrollment terminal

 crl configure



Phone-Proxy configuration

tls-proxy ASA-tls-proxy

 server trust-point _internal_PP_ctl_phoneproxy_file

!

ctl-file ctl_phoneproxy_file

 record-entry cucm-tftp trustpoint phoneproxy_trustpoint address 192.168.2.227

! the following entry is optional (for use with CAPF w/ LSC only phones

 record-entry capf trustpoint capf_trustpoint address 192.168.2.227

 no shutdown

!

media-termination mymta

 address 208.87.143.229 interface outside

 address 172.18.0.182 interface inside

!

phone-proxy ASA-phone-proxy

 media-termination mymta

 tftp-server address 172.18.2.26 interface inside

 tls-proxy ASA-tls-proxy

 ctl-file ctl_phoneproxy_file

 no disable service-settings



FW basic configuration (ACLs and NAT)

! public address of TFTP server

static (inside,outside) 192.168.2.227 172.18.2.26 netmask 255.255.255.255

!

! the only ACL entry required

access-list list outside permit tcp any host 192.168.2.227 eq tftp

!

object-group type service CUCM-PROXY-PORTS

 service-object tcp 2443

 service-object tcp 5061

 service-object tcp 3804

 service-object udp tftp

 service-object udp range 1024 65535

!

access-list phone-proxy-traffic permit object-group CUCM-PROXY-PORTS any host 192.168.2.227 

!

! Outside NAT forces return traffic from CUCM to go through ASA-PP

! Without this traffic from CUCM will take natural 0’s route to Internet causing asymmetrical behavior

nat (outside) 555 access-list phone-proxy-traffic outside

global(inside) 555 interface

!

class-map sec_sip

 match port tcp eq 5061

class-map sec_sccp

 match port tcp eq 2443

!

!

policy-map voice_policy

 class sec_sccp

  inspect skinny phone-proxy ASA-phone-proxy 

 class sec_sip

  inspect sip phone-proxy ASA-phone-proxy 

!

service-policy voice_policy interface outside

Cisco UCS: Tracking layer-2 paths

Basic tenets of network design:
Dual Nexus 7018s in the access layer
Dual UCS 6100 fabric interconnnects
ESXi 5.0 - hosts are dual attached to A and B side of fabric

We are tracking down an IP address in VLAN 301.

Please note the MAC address tables on the Nexus 7018s, the UCS 6100 (A & B), and Nexus 1000V… the following steps illustrate the well-known method used:

Nexus-7018-01# show ip arp vlan 301
IP ARP Table
Total number of entries: 2
Address         Age       MAC Address     Interface
192.168.247.66   00:06:57  0026.980c.76c1  Vlan301
192.168.247.70   00:01:18  0050.8888.6238  Vlan301


Nexus-7018-01# show mac address-table vlan 301
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 301      0026.980c.9bc1    static       -       F    F  sup-eth1(R)
* 301      0010.18a5.d9e0    dynamic   120        F    F  Po1
* 301      0026.980c.76c1    dynamic   450        F    F  Eth12/3
* 301      0050.8888.6237    dynamic   90         F    F  Eth12/3
* 301      0050.8888.6238    dynamic   90         F    F  Eth12/3 -> trunk to Nexus-7018-02
* 301      0050.8888.6239    dynamic   120        F    F  Eth12/7
* 301      547f.ee33.eb55    dynamic   120        F    F  Eth12/3
* 301      547f.ee35.cd95    dynamic   120        F    F  Eth12/7

Here we check the L2 adjacency table on Nexus-7018-02:

Nexus-7018-02# show mac address-table vlan 301
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
...
* 301      0050.8888.6238    dynamic   90         F    F  Eth12/7 -> UCS6100 - UCS-6100-01 -A 
...

SSH to UCS 6100s first, then "connect nxos" to enter Nexus like CLI directly on the fabric.

UCS-6100-01-B(nxos)# show mac address-table vlan 301
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY    Ports
---------+-----------------+--------+---------+------+----+------------------
* 301      0050.8888.6239    dynamic   20         F    F  Veth934

(the MAC address does not appear on Fab B), we could have bypassed and gone directly to Fab A based on CDP information on Nexus-7018-02.)

UCS-6100-01-A(nxos)# show mac address-table vlan 301
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY    Ports
---------+-----------------+--------+---------+------+----+------------------
* 301      0050.8888.6237    dynamic   0          F    F  Veth957
* 301      0050.8888.6238    dynamic   0          F    F  Veth961 -> MAC address learnt here -> note the association on CDP neighbor on the 1000V down a few steps  below…

We now check the mac-address table on the Nexus 1000V:

n1k-lab01# show mac address-table vlan 301
VLAN      MAC Address       Type    Age       Port                           Mod
---------+-----------------+-------+---------+------------------------------+---
...
301       0050.8888.6238    static  0         Veth20                         -> this reveals MAC address 
on VM guest
...


This leads us to Veth20 - we check the Nexus 1000V again:

n1k-lab01# show interface virtual pinning module 6

------------------------------------------------------
Veth      Pinned        Associated PO List of
          Sub Group id  interface     Eth interface(s)
------------------------------------------------------
Veth17    0             Po4           Eth6/1
Veth19    1             Po4           Eth6/2
Veth20    2             Po4           Eth6/3 -> this indicates VEM 4 connected over Po4

How to manually pin traffic:

n1k-lab01(config-if)# where
  conf; interface Vethernet20      admin@n1k-lab01
n1k-lab01(config-if)# pinning ?
  id  Configure sub-group ID for pinning

n1k-lab01(config-if)# pinning id ?
    Enter sub-group ID

n1k-lab01(config-if)# pinning id 1

MPLS Traffic Engineering Configuration

MPLS has been aptly defined as a layer "2.5" protocol. True to its roots, MPLS provides traffic engineering (TE) capabilities usually associated with L2 technologies such as ATM and Frame Relay, to L3 networks. Incorporating MPLS TE into an existing SP or large enterprise network can vastly improve the utilization of WAN links while also spreading out risk over them. In fact, this is the basis of how our largest IP backbones operate today.

In order to configure TE, the first step is to define a /32 TE loopback address on each router that is participating in the TE domain. Here, we have selected 10.88.47.x (clever? 0x8847 is the ether-type associated with MPLS unicast).

interface Loopback8847
description MPLS TE
ip address 10.88.47.1 255.255.255.255
ip ospf 1 area 0
end

Note that we have placed the interface in OSPF area 0 which has been prepared to participate the TE domain as the IGP.

router ospf 1
mpls ldp sync
mpls ldp autoconfig
mpls traffic-eng router-id Loopback8847
mpls traffic-eng area 0
router-id 10.88.47.1
log-adjacency-changes
network 10.4.1.2 0.0.0.0 area 0
!

We have to ensure that RSVP is configured on all interfaces that will participate in the TE domain. RSVP signalling with the use of TE extensions enables the head-end router to provision tunnels to the destination.

interface Gi0/0
ip rsvp bandwidth

Next, we configure the TE tunnel itself:

interface Tunnel9001
description MPLS TE tunnel to R6
ip unnumbered Loopback8847
ip ospf interface-retry 0
tunnel destination 10.88.47.6
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng priority 3 3
tunnel mpls traffic-eng bandwidth 16
tunnel mpls traffic-eng path-option 1 dynamic
tunnel mpls traffic-eng fast-reroute
no routing dynamic
end

Notice the IGP interaction with MPLS TE when you examine the new type 10 "opaque-area" LSAs under OSPF. These LSAs are flooded throughout the area.

Type-10 Opaque Link Area Link States (Area 0)

Link ID ADV Router Age Seq# Checksum Opaque ID
1.0.0.0 10.88.47.1 824 0x8000001F 0x006688 0
1.0.0.0 10.88.47.5 760 0x80000027 0x0007B9 0
1.0.0.0 10.88.47.6 626 0x80000022 0x008EAC 0
1.0.0.1 10.88.47.1 571 0x80000023 0x00A30E 1
1.0.0.1 10.88.47.5 1016 0x8000001F 0x009239 1

The rich texture of the LSAs is revealed here:

LS age: 781
Options: (No TOS-capability, DC)
LS Type: Opaque Area Link
Link State ID: 1.0.0.0
Opaque Type: 1
Opaque ID: 0
Advertising Router: 10.88.47.5
LS Seq Number: 80000027
Checksum: 0x7B9
Length: 140
Fragment number : 0

MPLS TE router ID : 10.88.47.5

Link connected to Point-to-Point network
Link ID : 10.88.47.6
Interface Address : 10.8.1.5
Neighbor Address : 10.8.1.6
Admin Metric : 1562
Maximum bandwidth : 8000
Maximum reservable bandwidth : 6000
Number of Priority : 8
Priority 0 : 6000 Priority 1 : 6000
Priority 2 : 6000 Priority 3 : 3000
Priority 4 : 2000 Priority 5 : 2000
Priority 6 : 2000 Priority 7 : 2000
Affinity Bit : 0x0
IGP Metric : 1562

Number of Links : 1

You can now verify MPLS - CEF interaction and confirm that the TE tunnels are up:

R1#show ip cef 172.31.254.6 detail
172.31.254.6/32, epoch 0, per-destination sharing
local label info: global/36
nexthop 10.88.47.6 Tunnel9002
nexthop 10.88.47.6 Tunnel9001

Finally, you can view the TE tunnel summary:

R1#show mpls traffic-eng tunnels summ
Signalling Summary:
LSP Tunnels Process: running
Passive LSP Listener: running
RSVP Process: running
Forwarding: enabled
Head: 2 interfaces, 2 active signalling attempts, 2 established
5 activations, 3 deactivations
0 SSO recovery attempts, 0 SSO recovered
Midpoints: 0, Tails: 0
Periodic reoptimization: every 3600 seconds, next in 2972 seconds
Periodic FRR Promotion: Not Running
Periodic auto-bw collection: every 300 seconds, next in 273 seconds

Verifying MPLS labels with CEF

Here is a quick and dirty way to verify that the expected labels are being imposed. This is especially useful within the SP core to verify MPLS LDP behavior and assist in troubleshooting, because we often have an inner and an outer label:

R6#show ip cef vrf blue serial 1/1:0 detail
IP CEF with switching (Table Version 23), flags=0x0
10 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 27
14 instant recursive resolutions, 0 used background process
96 leaves, 88 nodes, 95448 bytes, 1498 inserts, 1402 invalidations
0 load sharing elements, 0 bytes, 0 references
universal per-destination load sharing algorithm, id DCD67B0C
6(2) CEF resets, 43 revisions of existing leaves
Resolution Timer: Exponential (currently 1s, peak 2s)
0 in-place/0 aborted modifications
refcounts: 23311 leaf, 23040 node

Table epoch: 0 (10 entries at this epoch)

Adjacency Table has 9 adjacencies
7 IPv4 adjacencies
2 IPv6 adjacencies
10.4.1.2/32, version 20, epoch 0, cached adjacency to Serial1/1:0
0 packets, 0 bytes
via 10.8.1.5, Serial1/1:0, 0 dependencies
next hop 10.8.1.5, Serial1/1:0
valid cached adjacency
172.16.100.0/24, version 22, epoch 0, cached adjacency to Serial1/1:0
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with Se1/1:0, point2point, tags imposed: {28 41}
via 172.31.254.1, 0 dependencies, recursive
next hop 10.8.1.5, Serial1/1:0 via 172.31.254.1/32
valid cached adjacency
tag rewrite with Se1/1:0, point2point, tags imposed: {28 41}

BGP MPLS VPN Example

I hope to explain to you, curious reader, the essence behind the technique to configure a RFC 2547 BGP/MPLS VPN.

First, we create "loopback" interfaces (Lo1000 used here) on all three routers used in this example.

Next, we enable MPLS forwarding on the interfaces that participate in the forwarding path. Please note that R5 is acting as the "P" router, and R1 and R2 are "PE/CE" routers. The VRF instance "blue" represents the CE network.

R6 interfaces is shown here as an example:

interface Serial1/1:0
bandwidth 64
ip address 10.8.1.6 255.255.255.0
encapsulation ppp
mpls ip
no fair-queue
end

Next, we enable IGP in the "P" core. Here we use OSPF and place the interfaces in Area 0. Notice the special LDP commands included under OSPF to facilitate interaction with MPLS LDP.

router ospf 1
mpls ldp sync
mpls ldp autoconfig
mpls ldp autoconfig area 0
log-adjacency-changes
network 10.4.1.1 0.0.0.0 area 0
network 10.8.1.0 0.0.0.255 area 0

Now, we can verify MPLS LDP operation with the following commands:

show mpls forwarding table

Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
25 Untagged 172.16.100.0/24 4872 Fa0/0 10.7.1.2
26 Untagged 172.30.3.0/24 12615289 Fa0/0 10.7.1.2
27 Pop tag 192.168.200.0/24 5800 Se1/1:0 point2point
28 Pop tag 172.31.254.1/32 10771 Fa0/1 10.4.1.2
30 Untagged 10.8.1.6/32 0 Se1/1:0 point2point
31 Pop tag 10.10.10.10/32 0 Fa0/1 10.4.1.2
32 Pop tag 172.31.254.6/32 9321 Se1/1:0 point2point


We now proceed to configure BGP with multi-protocol extensions as follows:

router bgp 65001
no synchronization
bgp log-neighbor-changes
neighbor 172.31.254.6 remote-as 65001
neighbor 172.31.254.6 update-source Loopback1000
no auto-summary
!
address-family vpnv4
neighbor 172.31.254.6 activate
neighbor 172.31.254.6 send-community extended
exit-address-family
!
address-family ipv4 vrf blue
redistribute connected
no synchronization
exit-address-family
!

Let's check that BGP has established a neighbor relationship and is receiving the routes in the VRF (blue):

R1#show ip bgp vpnv4 vrf blue
BGP table version is 5, local router ID is 200.200.200.200
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:10 (default for vrf blue)
*> 172.16.100.0/24 0.0.0.0 0 32768 ?
*>i172.16.106.0/24 172.31.254.6 0 100 0 ?

Note the RT extended community:

R1#show ip bgp vpnv4 vrf blue 172.16.106.0
BGP routing table entry for 100:10:172.16.106.0/24, version 5
Paths: (1 available, best #1, table blue)
Not advertised to any peer
Local
172.31.254.6 (metric 1564) from 172.31.254.6 (192.168.91.6)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:100:10
mpls labels in/out nolabel/17

Note the MPLS label imposition - there are two labels imposed:

R1#show mpls forwarding-table vrf blue 172.16.106.0 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
None 17 172.16.106.0/24[V] \
0 Gi0/0 10.4.1.1
MAC/Encaps=14/22, MRU=1496, Label Stack{32 17}
00011FBE98010013C32C2CD88847 0002000000011000
VPN route: blue
No output feature configured

Finally, we can verify that we have connectivity:

R1#ping vrf blue 172.16.106.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.106.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/12 ms
R1#
1340399: Jan 20 10:01:36: MPLS les: Gi0/0: rx: Len 118 Stack {41 0 254} - ipv4 data
1340400: Jan 20 10:01:36: ICMP: echo reply rcvd, src 172.16.106.6, dst 172.16.100.1

You may wonder why R1 is receiving label 41. We need to examine R6 for an explanation:

R6#show mpls forwarding-table vrf blue 172.16.100.0 detail
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
None 41 172.16.100.0/24 0 Se1/1:0 point2point
MAC/Encaps=4/12, MRU=1496, Tag Stack{28 41}
FF030281 0001C00000029000
No output feature configured
Per-packet load-sharing

Configuring a basic MPLS VPN

MPLS – basic configuration with IOS

MPLS – sounds overwhelming? Well, it shouldn't be! Here how simple it can be:

Let’s consider a very basic but fully functional RFC 2547 BGP MPLS VPN. Here we have two routers R1 and R5, and both run Cisco IOS (12.4).

R1 [Gi0/0] --- [Fa0/1] R5

The first step is to enable “mpls ip”

Next, it’s a good idea to specify LDP as the label protocol (it is the default, but we do this for “completeness”)

R5#show run | i mpls
mpls label protocol ldp

We also have to enable “mpls ip” under the interfaces that will particpate (Gi0/0 and Fa0/0 in our case).

Now, let’s work on the VRF – we’ll call it “blue” here in our example:

ip vrf blue
rd 100:10
route-target export 100:10
route-target import 100:10

The syntax varies with a 2821 router:

vrf definition blue
rd 100:10
!
address-family ipv4
route-target export 100:10
route-target import 100:10
exit-address-family
!

At this point the LDP adjacency should form, but it didn’t! What happened?

On R5, LDP associated itself with a Loopback address that was not reachable from R1. Therefore we specify a LDP router-id:
R5(config)#mpls ldp router-id fastEthernet 0/1 force

R5#show mpls ldp discovery
Local LDP Identifier:
10.4.1.1:0
Discovery Sources:
Interfaces:
FastEthernet0/0 (ldp): xmit
FastEthernet0/1 (ldp): xmit/recv
LDP Id: 200.200.200.200:0

R5#show mpls ldp neigh
Peer LDP Ident: 10.4.1.2:0; Local LDP Ident 10.4.1.1:0
TCP connection: 10.4.1.2.27911 - 10.4.1.1.646
State: Oper; Msgs sent/rcvd: 40/46; Downstream
Up time: 00:00:14
LDP discovery sources:
FastEthernet0/1, Src IP addr: 10.4.1.2
Addresses bound to peer LDP Ident:
10.4.1.2 10.4.1.4 172.16.1.2 192.168.100.1
10.10.10.254 192.168.254.1 192.168.254.193 10.10.10.10
10.0.0.1 10.200.200.1 13.0.0.1 192.168.10.1
192.168.11.11 12.0.0.1 200.200.200.200 10.5.13.1
16.0.0.1 172.16.50.1 10.4.1.3 10.88.47.1

Ah! We have an adjacency now.

R1#show mpls ldp neighbor
Peer LDP Ident: 10.4.1.1:0; Local LDP Ident 10.4.1.2:0
TCP connection: 10.4.1.1.646 - 10.4.1.2.27911
State: Oper; Msgs sent/rcvd: 46/40; Downstream
Up time: 00:00:07
LDP discovery sources:
GigabitEthernet0/0, Src IP addr: 10.4.1.1
Addresses bound to peer LDP Ident:
10.7.1.1 10.7.1.5 10.4.1.1 172.16.5.5
10.9.1.1 192.168.55.5 55.0.0.55 60.0.0.5
172.16.50.5 1.1.1.1

We can now check LDP bindings:

Show mpls ldp bindings
Show mpls forwarding-table

Next, we configure BPG vpnv4 address-family thusly:

address-family vpnv4
neighbor 10.4.1.1 activate
neighbor 10.4.1.1 send-community extended
exit-address-family
!
address-family ipv4 vrf blue
redistribute connected
no synchronization
network 172.16.100.0 mask 255.255.255.0
exit-address-family
!

Here we verify the BGP speakers are exchanging VRF blue routes:

R1#show ip bgp vpnv4 vrf blue
BGP table version is 7, local router ID is 200.200.200.200
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:10 (default for vrf blue)
*> 172.16.100.0/24 0.0.0.0 0 32768 i
*> 172.16.105.0/24 10.4.1.1 0 0 65005 i

We can examine the labels:
R1#show ip bgp vpnv4 vrf blue labels
Network Next Hop In label/Out label
Route Distinguisher: 100:10 (blue)
172.16.100.0/24 0.0.0.0 22/nolabel(blue)
172.16.105.0/24 10.4.1.1 nolabel/27

Let's debug and observe:

debug mpls packet

R1#
1281051: Jan 16 13:17:27: MPLS les: Gi0/0: rx: Len 118 Stack {22 0 255} - ipv4 data
1281052: Jan 16 13:17:27: MPLS les: Gi0/0: rx: Len 118 Stack {22 0 255} - ipv4 data
1281053: Jan 16 13:17:27: MPLS les: Gi0/0: rx: Len 118 Stack {22 0 255} - ipv4 data
1281054: Jan 16 13:17:27: MPLS les: Gi0/0: rx: Len 118 Stack {22 0 255} - ipv4 data
1281055: Jan 16 13:17:27: MPLS les: Gi0/0: rx: Len 118 Stack {22 0 255} - ipv4 data

RSPAN capture with VACL redirect

This is a "new" instrumentation techinque on IOS that combines RSPAN and VACL, with the "redirect" feature coming into play.

! Create an extended ACL to match traffic that needs analysis
ip access-list extended traffic_to_analyze
permit tcp any any eq telnet
permit tcp any eq telnet any
!
! Create new RSPAN VLAN to copy traffic before we filter and capture
vlan 901
remote-span
!
! Define traditional SPAN monitor sessions to mirror all traffic to RSPAN VLAN
Monitor session 3 source interface vlan 100 both
Monitor session 3 destination remote vlan 901
!
! specify SPAN session details - Gi1/24 here goes to an analyzer
monitor session 4 source vlan 901
monitor session 4 destination interface Gi1/24
!
! Define VLAN access map with action redirect which bypasses traditional L2 forwarding
vlan access-map filter1 10
match ip address traffic_to_analyze
action redirect Gi1/24
!
! Apply VACL to RSPAN VLAN
vlan filter filter1 vlan-list 901
!

That's it! You can take a deep breath and relax knowing fully well that all the action is occurring on a copy of the traffic (in VLAN 901 here).

Note that although that this feature is supported generally with 12.2SX releases, this example was tested under 12.2(33)SXH2a

OTV in a nutshell

OTV edge devices (7018s) have two interfaces:
1. Internal facing L2 that functions exactly like any traditional switch with (mac, i/f) mappings in the table.
2. Core facing "join-interface" which connects to the "overlay-network"

Each site has an AED (active edge device) that is responsible for advertising (mac, join-interface-IP) into the overlay-network. All this happens over an multicast control-group (say, 239.1.1.1) that all the AED's join (core-facing, therefore the need for ASM capable core, which we have and support). IS-IS is the underlying link-state protocol which allows the AEDs to advertise their local mac-addresses that they learn from the "internal-facing" L2 segments.

Packet forwarding proceed thusly:

1. If dst mac is local to site, traditional switching with (mac, i/f) path is taken.
2. if dst mac is remote, OTV encapsulates complete L2 header (w/ 802.1q tag but strips off preamble and FCS) into an IP/UDP packet with dst-IP being the remote-AED that advertised the mac. The source is local-AED join-interface IP, and UDP dest port is 8472.
3. Multicast traffic handling is more sophisticated with (DS, DG) where (S,G) is mapped to (DS, DG) and fired off into core network. DS = Delivery Source, DG = Delivery Group. S = Source, G = Group.

VM vMotion:

1. When a VM moves, the mac moves with it.
2. The AED at the new site detects the presence of a mac on the outside (mac, remote-AED-IP) AND now on the inside (mac, i/f)
3. This triggers the AED to advertise the mac into the overlay with metric 0
4. The original site, upon noting this advertisement immediately withdraws its own (mac, join-interface-IP) with a rapid IS-IS LSP on the control-plane.

Cisco IOS Easy VPN

Cisco IOS Easy VPN De-mystified

At first, there seems to be nothing “easy” about configuring “Easy VPN” for IOS! But, don’t be alarmed, tenacious technician!

The critical thing to grasp is that this method requires IKE “group2” Diffie-Hellman exchange for a 1024bit modulus. Your VPN server has to have a IKE policy with this enabled.

Also, on the EZ client side, you’ll notice that the ISAKMP policies starting with priority 65515 are used from the client’s perspective. The first one specifies AES, SHA, and Group2. I recommend that you specify a matching policy on the server side first.

The server side configuration is classic remote access VPN type.

You will begin with a “crypto isakmp client configuration group ” and work your way to “crypto isakmp profile” and “crypto ipsec profile” where all the elements are tied together, so that they can be applied to the dynamic map or virtual-template.

Consider:

crypto isakmp client configuration group iosvpn
key cisco50
pool iosvpnpool
acl 144
banner ^CWelcome to R6 IOS remote access VPN ^C

crypto isakmp profile iosvpn
match identity group iosvpn
client authentication list LOCO
isakmp authorization list LOCO
client configuration address respond
virtual-template 20

crypto ipsec transform-set ccie esp-3des esp-sha-hmac

crypto ipsec profile iosvpn
set transform-set ccie
set reverse-route tag 23501
set isakmp-profile iosvpn

aaa new-model
aaa authentication login LOCO local
aaa authorization network LOCO local

ip local pool iosvpnpool 192.168.99.1 192.168.99.10

DMVPN through firewalls in virtual multi-context mode

When you have an ASA running in multi-context mode, you have to pay extra attention to the flow classifier.

Consider:

ccie-asa1/india# packet-tracer input out icmp 192.168.70.1 8 0 10.7.1.2
Result:input-interface: outinput-status: upinput-line-status: upAction: dropDrop-reason: (ifc-classify) Virtual firewall classification failed

Let's work backwards. Here, we are dealing with an external network 192.168.70.0 /24 trying t reach a internal network 10.7.1.0/24 in a "no nat-control" setup. The multi-context ASA here has multiple contexts sharing an outside interface (common in SP and hosting environments).

Here is the challenge:
When a packet arrives at the outside interface destined for a destination inside one of the contexts, how does the firewall know which one to forward to? Without unique mac-addresses, the layer 2 address does not give any clues. The firewall relies on NAT configuration to determine the context.

For example:
In the "ccie-asa1/india" context
static (inside,out) 10.7.1.1 10.7.1.1 netmask 255.255.255.255

Now, the classifier has information to act on the packet is sent to the "india" context.

This behavior is critical to grasp when it comes to configuring DMVPN through the firewall. Here, we have a need to allow IP/47 (GRE) traffic to the 10.7.1.1 address, or IP/50 (ESP) or IP/51 (AH). You can be very specific and allow just the protocol you are using. Don't forget udp/500 and if necessary, udp/4500 for IKE.