OCI FastConnect, in depth: circuits, BGP, path selection and redundant design

The connection at a glance

Three ways to connect

There are three ways to get a circuit, and they differ in who owns the physical link:

The partner model is fastest to set up because the physical work is done; the partner handles part of the BGP configuration depending on whether it offers layer 2 or layer 3 service. The other two models give you a dedicated port and direct control of BGP, and are what you pick for high bandwidth or encryption at layer 2.

The building blocks

A FastConnect setup is built from a small set of objects, and knowing which is which makes the console and the error messages readable:

• Cross-connect: one physical port on an Oracle device at a FastConnect location, patched to your equipment.
• Cross-connect group: a link aggregation group (LAG) of one or more cross-connects, up to 8 wide. You set minimum links, the number of member links that must be up for the group to stay up; it defaults to 1.
• Letter of Authorization (LOA): the document the facility needs to run the patch. It is valid for 60 days and can be extended by 30 days in the console. An expired LOA is a common reason a cross-connect order stalls.
• Virtual circuit: an isolated logical connection over the cross-connect group, carried on its own 802.1Q VLAN with its own BGP session. A single group can carry several virtual circuits.
• DRG: the gateway a private virtual circuit terminates on, which routes onwards to attached VCNs.

Each virtual circuit is private peering or public peering. Private peering carries RFC 1918 traffic between your network and your VCNs through a DRG, as if OCI were another site on your WAN. Public peering reaches OCI public services, such as Object Storage, using public addresses but without crossing the internet. For public peering you advertise public prefixes you own, which Oracle verifies, and a routing policy controls how widely Oracle advertises them: ORACLE_SERVICE_NETWORK, REGIONAL, MARKET_LEVEL or GLOBAL.

Public peering is not the only way to reach Oracle services privately. If you already have private peering, the DRG can route on-premises traffic into a VCN and through its service gateway to Object Storage and other services, keeping everything on private addressing and under your VCN route rules. Public peering suits networks that want to reach those services from their own public address space without a VCN in the path; private peering through a service gateway suits teams who prefer one set of routing and security rules.

The physical layer

On a dedicated port, the physical layer has to match Oracle's requirements exactly, or the link never comes up:

• Single-mode fibre with duplex LC connectors. Optics: 1000BASE-LX for 1 Gbps (auto-negotiation off), 10G LR for 10 Gbps, 100GBASE-LR4 QSFP28 for 100 Gbps and 400GBASE-LR4 QSFP-DD for 400 Gbps, each with a published receive-power window.
• LACP with short timers (3 packets at 1-second intervals) for cross-connect groups. A single non-LAG cross-connect is possible if your router cannot do LAG, but it gives up member-level redundancy.
• 802.1Q single tagging, with a VLAN you choose from 100 to 4094.
• A maximum IP MTU of 9000 (interface MTU 9196 including the frame check sequence). Each virtual circuit is created with MTU_1500 or MTU_9000; jumbo frames only help if every hop on both sides uses them.
• An optional interface hold timer, configurable from 0.5 to 3 seconds in half-second steps, which delays declaring a flapping interface down.

BGP on a virtual circuit

Every virtual circuit runs BGPv4. Oracle's side uses ASN 31898 in commercial regions (Serbia Central uses 14544). Your side can use a 2-byte or 4-byte ASN. Private ASNs from 64512 to 65533 are allowed, but 65534 is reserved and cannot be used; public peering requires a public ASN. Other rules to plan around:

• For private peering you pick the point-to-point addresses, from a /28 down to a /31, one pair per virtual circuit. For public peering Oracle assigns them.
• MD5 authentication is optional; Oracle supports keys up to 128 bits.
• Oracle's default keepalive is 10 seconds and hold time 30 seconds, adjustable within 6 to 60 and 18 to 180 seconds. With default timers a silent failure takes up to 30 seconds to detect, so enable Bidirectional Forwarding Detection (BFD) for sub-second detection.
• Prefix limits: 200 prefixes on a public virtual circuit, and 2,000 IPv4 plus 500 IPv6 prefixes on a private one. Exceeding the limit can bring the session down, so summarise.

Here is a private virtual circuit created with the OCI CLI and the matching session on an FRRouting router. The addresses, VLAN and ASN are examples:

oci network virtual-circuit create \
  --compartment-id "$COMPARTMENT_OCID" --type PRIVATE \
  --display-name dc1-to-oci-a --bandwidth-shape-name "10 Gbps" \
  --gateway-id "$DRG_OCID" --customer-asn 64601 \
  --ip-mtu MTU_9000 --is-bfd-enabled true \
  --cross-connect-mappings '[{
      "crossConnectOrCrossConnectGroupId": "'"$CCG_A_OCID"'",
      "customerBgpPeeringIp": "10.255.0.2/30",
      "oracleBgpPeeringIp":   "10.255.0.1/30",
      "vlan": 210, "bgpMd5AuthKey": "'"$BGP_KEY"'"}]'

! FRRouting, router A, VLAN 210 sub-interface carries 10.255.0.2/30
router bgp 64601
 neighbor 10.255.0.1 remote-as 31898
 neighbor 10.255.0.1 password <same key as bgpMd5AuthKey>
 neighbor 10.255.0.1 timers 10 30
 neighbor 10.255.0.1 bfd
 address-family ipv4 unicast
  network 10.20.0.0/16            ! summarise: one route, not hundreds
  neighbor 10.255.0.1 prefix-list TO-OCI out
  neighbor 10.255.0.1 prefix-list FROM-OCI in
 exit-address-family

How Oracle chooses a path

Once routes are exchanged, two questions decide where traffic goes. The first is ordinary IP routing: the most specific matching prefix wins. The second applies when the same prefix arrives over several connections. Oracle attaches different AS path lengths by connection type and then prefers the shortest:

So a VPN advertising the same prefix as FastConnect is automatically a standby for traffic leaving OCI. Oracle picks its return path by AS path regardless of which path the connection arrived on, so routing can be asymmetric. Stateful firewalls on your side drop asymmetric flows, so control both directions. Outbound from your network, use local preference on your routers. Inbound to your network, prepend your own ASN on the path you want Oracle to treat as standby, and keep the ranking strict. With FastConnect circuits A and B plus a BGP VPN: leave A at length 1, prepend your ASN once on B (length 2), and prepend at least twice on the VPN advertisement (length 4 or more) so it ranks last even after A fails. If B were prepended to length 3, it would lose to an unprepended BGP VPN at length 2.

If you want both circuits active instead, advertise the same prefixes with equal AS path length and enable ECMP on the DRG route table that imports those routes; ECMP is off by default, applies to several FastConnect virtual circuits or several IPSec tunnels but not a mix, and spreads traffic per flow, so one large transfer still uses one circuit. The general BGP mechanics are explained in BGP for engineers.

Designing for failure, with a worked example

A single circuit is a single point of failure, including the Oracle device, your router, the patch and the facility. Oracle recommends device redundancy in all cases, and location redundancy where a region has more than one FastConnect location. A sound design is two cross-connect groups on different Oracle devices, ideally at two locations, each to its own router, plus a Site-to-Site VPN as a last resort.

Worked example. A company runs a nightly replication of 40 TB from its data centre to OCI and steady daytime traffic peaking at 6 Gbps. At a 10 Gbps line rate, 40 TB takes 40e12 x 8 / 10e9 = 32,000 seconds, about 8.9 hours; at a realistic 80% efficiency it is about 11 hours, which misses an 8-hour window. Two 10 Gbps circuits used active-active finish in about 5.6 hours. But if one fails, the survivor carries everything, the job runs 11 hours, and it spills into the 6 Gbps daytime peak. The options are a 100 Gbps port, or two 10 Gbps circuits plus a throttle that halves the replication rate whenever one BGP session is down. Size for the failure case, not the normal case: each circuit should carry the critical load alone.

Cost follows the port, not the traffic. Dedicated FastConnect is billed per port-hour by speed, and Oracle charges nothing for data transferred over it in either direction; partner and carrier fees are separate. That makes FastConnect cheaper than internet egress for heavy transfers, a contrast with AWS Direct Connect, which bills per gigabyte out.

Encryption

FastConnect is private but not encrypted by default. Two options exist. MACsec encrypts at layer 2 between your router and Oracle's device; Oracle announced it for 10 and 100 Gbps dedicated connections and LAGs, and you need MACsec-capable hardware and a key managed in OCI Vault. Alternatively, run IPSec over FastConnect: a Site-to-Site VPN whose tunnels ride the private circuit. The CLI's --is-transport-mode flag creates a virtual circuit meant to carry only encrypted traffic. IPSec works with any provisioning model but costs throughput per tunnel. Pair either with the access controls in OCI IAM so only network admins can change circuits.

Operating it

• Watch BGP state and the light. Alert on any BGP session leaving Established and on receive power drifting towards the edge of the optics window; a dirty connector shows up there first.
• Test failover on a schedule. Shut one session in a maintenance window and confirm traffic moves within the BFD detection time and nothing breaks on asymmetric return paths.
• Drain before maintenance. Virtual circuits accept a --traffic-mode DRAIN setting so traffic leaves gracefully before you touch a link.
• Track prefix counts. A route leak from your side can trip the prefix limit and drop the session; filter outbound with explicit prefix lists.
• Renew LOAs early. Calendar the 60-day expiry for every pending cross-connect.

Failure modes

What to do next

1. Choose the model: partner for speed of setup, colocation or third-party for 100 or 400 Gbps, dedicated control or MACsec.
2. Size each circuit to carry the critical load alone, using the failure-case arithmetic above.
3. Order two cross-connect groups on separate devices, and separate locations where the region offers them.
4. Plan ASN, VLANs, /30 or /31 peering addresses and MTU before ordering; record them per virtual circuit.
5. Enable BFD, set MD5 keys, and filter both directions with prefix lists.
6. Add a Site-to-Site VPN as a standby, prepend it more than any FastConnect backup so it always ranks last, and enable DRG ECMP only if you want circuits active-active.
7. Set alerts on BGP state and optics, and run a failover test before going live.