BGP (Border Gateway Protocol) between VMware NSX-T and USG (Ubiquiti Security Gateway)
Setup exterior BGP (Border Gateway Protocol) between VMware NSX-T and USG (Ubiquiti Security Gateway). When trying NSX-T most start with static routes. If we want to experiment with more advanced failure recovery modes with the edge, we need to move from static routes to a dynamic routing protocol between the upstream router and the NSX-T edge nodes. This exercise configures BGP between NSX-T and a USG.
Sample Homelab Topology
Values in this example:
USG 172.16.31.1
NSX-T T0 172.16.31.2
NAT for Infrastructure 172.16.30.0/24
PKS SNAT VIP Pool 172.16.29.0/24The Unifi will be in BGP AS 64512. NSX-T will use BGP AS 64513. BGP Autonomous System Numbers (AS) 64512-65534 are reserved similar to how RC1912 addresses are reserved for private use.
Configuting NSX-T
1) In NSX Manager enable BGP. Select the Tier 0 route and goto the the Routing / BGP page.
Enable BGP. This environment only has one router and one edge node, therefore ECMP (equal cost multipathing) is disabled and Graceful Restart is enabled. Enter 64513 as the AN. Keep Alive and Hold Down times at their defaults.
In this example, we added route aggregation for 172.16.28.0/24, 172.16.29.0/24, and 172.16.30.0/24 to make the route table smaller. This is optional.
2) Add the USG as a BGP Neighbor. Click “+ADD” under Neighbors. Enter 64512 as the remote AS and the IP address of the upstream router. We already have a static default route pointing to this router when we setup NSX-T.
Next configure the USG.
Configuring the USG
Our USG is managed by the Unifi Controller. THe BGP settings are not configurable in Controller yet. Here is the JSON needed in config.gateway.json
"protocols": {
"bgp": {
"64512": {
"neighbor": {
"172.16.31.2": {
"remote-as": "64513"
}
},
"parameters": {
"router-id": "172.16.31.1"
},
"redistribute": {
"static": "''"
}
}
}
}Making that change and reprovisioning the router will enable BGP.
OPTIONAL Developing a different BGP configuration
OPTIONAL To try different configurations rapidly before changing config.gateway.json, use the CLI. These commands are from Cowger’s blog post on enabling iBGP referenced at the end.
admin@usg:~$ configure
set[edit]
admin@usg# set protocols bgp 64512 parameters router-id 172.16.31.1
[edit]
admin@usg# set protocols bgp 64512 neighbor 172.16.31.2 remote-as 64513
[edit]
admin@usg# set protocols bgp 64512 redistribute static
[edit]
admin@usg# commit
[ protocols bgp 64512 ]
Starting routing daemon: bgpd.
[edit]
admin@usg# save
Saving configuration to '/config/config.boot'...
Done
[edit]
admin@usg# exitView the current configuration and look for the BGP section to see what needs to be added to config.gateway.json.
admin@usg:~$ mca-ctrl -t dump-cfgViewing the USG Results
Check BGP on the USG.
With Route Summarization
$ show ip bgp
BGP table version is 0, local router ID is 172.16.31.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 172.16.29.0/24 172.16.31.2 0 64513 ?
*> 172.16.30.0/24 172.16.31.2 0 64513 ?
Total number of prefixes 2Without Route Summarization
If you did not use Route Summarization when enabling BGP, the route list will be longer:
admin@usg:~$ show ip bgp
BGP table version is 0, local router ID is 172.16.31.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 172.16.29.10/32 172.16.31.2 0 64513 ?
*> 172.16.29.11/32 172.16.31.2 0 64513 ?
*> 172.16.29.12/32 172.16.31.2 0 64513 ?
*> 172.16.29.13/32 172.16.31.2 0 64513 ?
*> 172.16.29.14/32 172.16.31.2 0 64513 ?
*> 172.16.29.20/32 172.16.31.2 0 64513 ?
*> 172.16.29.21/32 172.16.31.2 0 64513 ?
*> 172.16.30.4/32 172.16.31.2 0 64513 ?
*> 172.16.30.6/32 172.16.31.2 0 64513 ?
*> 172.16.30.7/32 172.16.31.2 0 64513 ?
*> 172.16.30.9/32 172.16.31.2 0 64513 ?Viewing the NSX-T Results
Check NSX-T for BGP:
- SSH into the edge node.
- Find the Tier 0 Logical Router.
- Select the router.
View the BGP Route table.
nsxe-1> get logical-routers Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 5b538d01-8110-4bc9-b1ca-cc6bc8ad1d77 1 7 DR-T1Router-PKS-Services DISTRIBUTED_ROUTER_TIER1 5 ae8c73cc-21d7-40b1-8051-a8bac9c1d198 2 2 SR-DefaultT0Router SERVICE_ROUTER_TIER0 5 33a5fcc7-eb2f-488b-83aa-6f37f3f796f5 3 3075 SR-lb-pks-93a1e808-3cc8-4410-b1 SERVICE_ROUTER_TIER1 5 a2197c71-7726-4173-8a18-0ae696a2657e 4 3073 DR-pks-93a1e808-3cc8-4410-b162- DISTRIBUTED_ROUTER_TIER1 5 3882ab42-d404-4166-a1ed-cc9ddf352681 5 3077 DR-pks-93a1e808-3cc8-4410-b162- DISTRIBUTED_ROUTER_TIER1 5 f35c8d0f-fe57-4d1e-a0dc-71122f18f363 6 4 DR-T1-Router-PAS-ERT DISTRIBUTED_ROUTER_TIER1 4 7934ad9c-e645-442b-bbe0-153bb191db7a 7 3079 DR-pks-93a1e808-3cc8-4410-b162- DISTRIBUTED_ROUTER_TIER1 5 ba2d177f-832e-4323-a190-b37cff45df9a 8 5 SR-T1-Router-PAS-ERT SERVICE_ROUTER_TIER1 5 3aca0b9b-2550-483e-b972-96a8c3d62085 9 1 DR-DefaultT0Router DISTRIBUTED_ROUTER_TIER0 14 68b93df9-b849-4246-813b-8946d8c2d1fd 10 6 DR-T1-Router-PAS-Services DISTRIBUTED_ROUTER_TIER1 5 b67d5a87-59be-4122-a724-28d66dbc82e8 11 3 DR-T1-Router-PCF-Infra DISTRIBUTED_ROUTER_TIER1 5 c3302b78-d6e8-4721-80e7-8bbde69589cb 12 3078 DR-pks-93a1e808-3cc8-4410-b162- DISTRIBUTED_ROUTER_TIER1 5 ca456647-90a2-47d8-8e43-0e52129b7090 13 3076 DR-pks-93a1e808-3cc8-4410-b162- DISTRIBUTED_ROUTER_TIER1 5 fa779916-877b-4457-beb8-9422a08634ee 14 3074 DR-lb-pks-93a1e808-3cc8-4410-b1 DISTRIBUTED_ROUTER_TIER1 4 nsxe-1> vrf 2 nsxe-1(tier0_sr)> get route bgp Flags: c - connected, s - static, b - BGP, ns - nsx_static nc - nsx_connected, rl - router_link, t0n: Tier0-NAT, t1n: Tier1-NAT t1l: Tier1-LB VIP, t1s: Tier1-LB SNAT b 172.16.29.0/24 [200/0] via Null b 172.16.30.0/24 [200/0] via Null