Arista EOS Layer3 EVPNとダイナミックルーティングの併用 – 自宅で体験できるファブリックネットワーク

2020.10.10
スポンサーリンク

Arista EOSでLayer3 EVPN環境で、インターネットや他拠点とダイナミックルーティングを設定する方法を説明します。

構成図

以下の環境で動作確認を行います。leaf01, leaf02がそれぞれisp01とisp02に接続されており、isp01とisp02がそれぞれ疎通可能になるようなシナリオについて考察します

                          +---------------+
      192.168.13.0/24  .3 |    spine03    | .3   192.168.23.0/24
              +-----------+Lo0:10.3.3.3/32+----------+
              |        e1 +---------------+ e2       |
              |                                      |
           e2 | .1                                e1 | .2
+-------------+----------------+       +-------------+----------------+ 
|            leaf01            |       |            leaf02            |
|       Lo0:10.1.1.1/32        |       |       Lo0:10.2.2.2/32        |
|  vlan100:192.168.100.254/24  |       |  vlan100:192.168.200.254/24  |
+-------------+----------------+       +-------------+----------------+
           e1 | vlan100                           e2 | vlan200
              |                                      |
           e1 | vlan100                           e1 | vlan200
+-------------+----------------+       +-------------+----------------+ 
|            isp01             |       |            isp02             |
|    vlan100:192.168.100.1/24  |       |    vlan200:192.168.200.2/24  |
|      Lo0:10.11.11.11/32      |       |      Lo0:10.22.22.22/32      |
+------------------------------+       +------------------------------+

初期設定

初期設定時点でisp01のvlan100とisp02のvlan200が互いに疎通可能な状態とします。

isp01のLo0とisp02のLo02は互いに疎通不能で、このLoopback同士が互いに接続可能になるようなダイナミックルーティングの設定について考察します。

isp01
! Command: show running-config
! device: isp01 (vEOS, EOS-4.25.0FX-LDP-RSVP)
!
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model ribd
!
hostname isp01
!
spanning-tree mode mstp
!
no aaa root
!
username admin role network-admin secret sha512 $6$1HdZV8EnhStZl7Jq$ykA.fAxEM2WTNGiPYNFMQ9a88CjY2yK9F0tZM1x8xMWUAcmOQsuEWHVvYHo2OBUnZcxw2aqn767XrqkLaV8CS/
!
vlan 100
!
vrf instance CONSOLE
!
interface Ethernet1
   switchport access vlan 100
!
interface Loopback0
   ip address 10.11.11.11/32
!
interface Management1
   vrf CONSOLE
   ip address 192.168.1.41/24
!
interface Vlan100
   ip address 192.168.100.1/24
!
ip routing
ip routing vrf CONSOLE
!
ip route 0.0.0.0/0 192.168.100.254
ip route vrf CONSOLE 0.0.0.0/0 192.168.1.1
!
end
leaf01
! Command: show running-config
! device: leaf01 (vEOS, EOS-4.25.0FX-LDP-RSVP)
!
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname leaf01
!
spanning-tree mode mstp
!
no aaa root
!
username admin role network-admin secret sha512 $6$8g4im0prH7WjskGr$WSWnk6NXojEYqo03GaaS0kg0RdAjkwKCF7hqpLk1b2/aIyjAUwIcBWcq.7zuWDyxdv8Otf74JsHzZnn9aR9hG.
!
vlan 100
!
vrf instance CONSOLE
!
vrf instance GATEWAY
!
interface Ethernet1
   switchport access vlan 100
!
interface Ethernet2
   no switchport
   ip address 192.168.13.1/24
!
interface Loopback0
   ip address 10.1.1.1/32
!
interface Management1
   vrf CONSOLE
   ip address 192.168.1.42/24
!
interface Vlan100
   vrf GATEWAY
   ip address 192.168.100.101/24
   ip virtual-router address 192.168.100.254
!
interface Vxlan1
   vxlan source-interface Loopback0
   vxlan udp-port 4789
   vxlan vlan 100 vni 90100
   vxlan vrf GATEWAY vni 9999
!
ip virtual-router mac-address 00:00:00:00:00:0a
!
ip routing
ip routing vrf CONSOLE
ip routing vrf GATEWAY
!
ip route vrf CONSOLE 0.0.0.0/0 192.168.1.1
!
router bgp 65000
   no bgp default ipv4-unicast
   timers bgp 10 30
   neighbor 10.2.2.2 remote-as 65000
   neighbor 10.2.2.2 update-source Loopback0
   neighbor 10.2.2.2 send-community
   neighbor 10.2.2.2 maximum-routes 12000
   !
   vlan 100
      rd 10.1.1.1:100
      route-target both 100:90100
      redistribute learned
   !
   address-family evpn
      neighbor 10.2.2.2 activate
   !
   vrf GATEWAY
      rd 10.1.1.1:9999
      route-target import evpn 99:9999
      route-target export evpn 99:9999
      redistribute connected
!
router ospf 1
   network 10.1.1.1/32 area 0.0.0.0
   network 192.168.13.0/24 area 0.0.0.0
   max-lsa 12000
!
end
spine03
! Command: show running-config
! device: spine03 (vEOS, EOS-4.25.0FX-LDP-RSVP)
!
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model ribd
!
hostname spine03
!
spanning-tree mode mstp
!
no aaa root
!
username admin role network-admin secret sha512 $6$XNluPtcOuBEDxR5H$EqK9UnbxO1KHRl.1tmHKZoRw9mFvxp3/hFJqQUDQ9wnkXjINlDPRUoZ4wPECNmKue.BfLxYDT/LtYLe8u6T4J0
!
vrf instance CONSOLE
!
interface Ethernet1
   no switchport
   ip address 192.168.13.3/24
!
interface Ethernet2
   no switchport
   ip address 192.168.23.3/24
!
interface Loopback0
   ip address 10.3.3.3/32
!
interface Management1
   vrf CONSOLE
   ip address 192.168.1.43/24
!
ip routing
ip routing vrf CONSOLE
!
ip route vrf CONSOLE 0.0.0.0/0 192.168.1.1
!
router ospf 1
   network 10.3.3.3/32 area 0.0.0.0
   network 192.168.13.0/24 area 0.0.0.0
   network 192.168.23.0/24 area 0.0.0.0
   max-lsa 12000
!
end
leaf02
! Command: show running-config
! device: leaf02 (vEOS, EOS-4.25.0FX-LDP-RSVP)
!
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model multi-agent
!
hostname leaf02
!
spanning-tree mode mstp
!
no aaa root
!
username admin role network-admin secret sha512 $6$bVpJ2.2SeukhvZ49$i20QxV1G6Y4RsuFV57vcPuvB2gZo5LsVHdE9Ht9yxzQoNaieMNPkBB6gdvbsykwBqXG2UDN2peT49dbeQteuX/
!
vlan 200
!
vrf instance CONSOLE
!
vrf instance GATEWAY
!
interface Ethernet1
   no switchport
   ip address 192.168.23.2/24
!
interface Ethernet2
   switchport access vlan 200
!
interface Loopback0
   ip address 10.2.2.2/32
!
interface Management1
   vrf CONSOLE
   ip address 192.168.1.44/24
!
interface Vlan200
   vrf GATEWAY
   ip address 192.168.200.102/24
   ip virtual-router address 192.168.200.254
!
interface Vxlan1
   vxlan source-interface Loopback0
   vxlan udp-port 4789
   vxlan vlan 200 vni 90200
   vxlan vrf GATEWAY vni 9999
!
ip virtual-router mac-address 00:00:00:00:00:0a
!
ip routing
ip routing vrf CONSOLE
ip routing vrf GATEWAY
!
ip route vrf CONSOLE 0.0.0.0/0 192.168.1.1
!
router bgp 65000
   no bgp default ipv4-unicast
   timers bgp 10 30
   neighbor 10.1.1.1 remote-as 65000
   neighbor 10.1.1.1 update-source Loopback0
   neighbor 10.1.1.1 send-community
   neighbor 10.1.1.1 maximum-routes 12000
   !
   vlan 200
      rd 10.2.2.2:200
      route-target both 200:90200
      redistribute learned
   !
   address-family evpn
      neighbor 10.1.1.1 activate
   !
   vrf GATEWAY
      rd 10.2.2.2:9999
      route-target import evpn 99:9999
      route-target export evpn 99:9999
      redistribute connected
!
router ospf 1
   network 10.2.2.2/32 area 0.0.0.0
   network 192.168.23.0/24 area 0.0.0.0
   max-lsa 12000
!
end
isp02
! Command: show running-config
! device: isp02 (vEOS, EOS-4.25.0FX-LDP-RSVP)
!
! boot system flash:/vEOS-lab.swi
!
transceiver qsfp default-mode 4x10G
!
service routing protocols model ribd
!
hostname isp02
!
spanning-tree mode mstp
!
no aaa root
!
username admin role network-admin secret sha512 $6$k0Y65l5FD1m2IXQu$PGCSozozxe9XwHS2jPp/iDTy11fm34zcPkJfb1.hY0Ajgzoa9GpmVjqAF8nW1I0P9Per1sLq6oSiEOGEaYLlf.
!
vlan 200
!
vrf instance CONSOLE
!
interface Ethernet1
   switchport access vlan 200
!
interface Loopback0
   ip address 10.22.22.22/32
!
interface Management1
   vrf CONSOLE
   ip address 192.168.1.45/24
!
interface Vlan200
   ip address 192.168.200.2/24
!
ip routing
ip routing vrf CONSOLE
!
ip route 0.0.0.0/0 192.168.200.254
ip route vrf CONSOLE 0.0.0.0/0 192.168.1.1
!
end

動作確認(1) 初期設定の確認

疎通確認

isp01とisp02の間でvlan interface同士は疎通可能であるものの、Loopback同士は疎通不能である事を確認します。

isp01#ping 192.168.200.2
PING 192.168.200.2 (192.168.200.2) 72(100) bytes of data.
80 bytes from 192.168.200.2: icmp_seq=1 ttl=62 time=20.6 ms
80 bytes from 192.168.200.2: icmp_seq=2 ttl=62 time=18.2 ms
80 bytes from 192.168.200.2: icmp_seq=3 ttl=62 time=18.1 ms
80 bytes from 192.168.200.2: icmp_seq=4 ttl=62 time=22.9 ms
80 bytes from 192.168.200.2: icmp_seq=5 ttl=62 time=23.2 ms

--- 192.168.200.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 60ms
rtt min/avg/max/mdev = 18.146/20.636/23.238/2.204 ms, pipe 3, ipg/ewma 15.109/20.787 ms
isp01#
isp01#
isp01#ping 10.22.22.22
PING 10.22.22.22 (10.22.22.22) 72(100) bytes of data.
From 192.168.100.101 icmp_seq=1 Destination Net Unreachable

--- 10.22.22.22 ping statistics ---
5 packets transmitted, 0 received, +1 errors, 100% packet loss, time 39ms
pipe 2
isp01#

ルーティング等の確認

leaf01とleaf02の間でルーティングとMACアドレスを互いに学習している事を確認します。

leaf01#show vxlan address-table 
          Vxlan Mac Address Table
----------------------------------------------------------------------

VLAN  Mac Address     Type     Prt  VTEP             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
4094  0050.5607.225b  EVPN     Vx1  10.2.2.2         1       0:02:12 ago
Total Remote Mac Addresses for this criterion: 1
leaf01#
leaf01#
leaf01#
leaf01#show bgp evpn 
BGP routing table information for VRF default
Router identifier 10.1.1.1, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 10.1.1.1:100 mac-ip 000c.2905.9ba5
                                 -                     -       -       0       i
 * >     RD: 10.1.1.1:100 mac-ip 000c.2905.9ba5 192.168.100.1
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 mac-ip 0050.56f4.e19e
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.2.2.2:200 mac-ip 0050.56f4.e19e 192.168.200.2
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:100 mac-ip 5555.5555.5555
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 mac-ip 5555.5555.5555
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:100 imet 10.1.1.1
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 imet 10.2.2.2
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:9999 ip-prefix 192.168.100.0/24
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:9999 ip-prefix 192.168.200.0/24
                                 10.2.2.2              -       100     0       i
leaf01#
leaf01#
leaf01#
leaf01#show bgp evpn 
BGP routing table information for VRF default
Router identifier 10.1.1.1, local AS number 65000
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup
                    % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  LocPref Weight  Path
 * >     RD: 10.1.1.1:100 mac-ip 000c.2905.9ba5
                                 -                     -       -       0       i
 * >     RD: 10.1.1.1:100 mac-ip 000c.2905.9ba5 192.168.100.1
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 mac-ip 0050.56f4.e19e
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.2.2.2:200 mac-ip 0050.56f4.e19e 192.168.200.2
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:100 mac-ip 5555.5555.5555
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 mac-ip 5555.5555.5555
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:100 imet 10.1.1.1
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:200 imet 10.2.2.2
                                 10.2.2.2              -       100     0       i
 * >     RD: 10.1.1.1:9999 ip-prefix 192.168.100.0/24
                                 -                     -       -       0       i
 * >     RD: 10.2.2.2:9999 ip-prefix 192.168.200.0/24
                                 10.2.2.2              -       100     0       i
leaf01#

動作確認(2) OSPFによる接続例

OSPFの設定

isp01とleaf01の間でOSPFを設定します。

# ips01
router ospf 1
   network 10.11.11.11/32 area 0.0.0.0
   network 192.168.100.0/24 area 0.0.0.0

# leaf01
router ospf 2 vrf GATEWAY
   redistribute bgp
   network 192.168.100.0/24 area 0.0.0.0

isp01とleaf01の間でOSPF neighborが確立された事を確認します。

leaf01#show ip ospf neighbor vrf GATEWAY
Neighbor ID     Instance VRF      Pri State                  Dead Time   Address         Interface
10.11.11.11     2        GATEWAY  1   FULL/DR                00:00:37    192.168.100.1   Vlan100

leaf01でOSPFとBGPの再配送を行います。

# leaf01
router ospf 2 vrf GATEWAY
   redistribute bgp
!
router bgp 65000
   vrf GATEWAY
      redistribute ospf

isp01がleaf02のプレフィックス(192.168.200.0/24)を学習している事を確認します。

isp01#show ip route

VRF: default
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort:
 S        0.0.0.0/0 [1/0] via 192.168.100.254, Vlan100

 C        10.11.11.11/32 is directly connected, Loopback0
 C        192.168.100.0/24 is directly connected, Vlan100
 O E2     192.168.200.0/24 [110/1] via 192.168.100.101, Vlan100

isp01#

leaf02がisp01のプレフィックス(10.11.11.11/32)を学習している事を確認します。

leaf02#show ip route vrf GATEWAY

VRF: GATEWAY
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort is not set

 B I      10.11.11.11/32 [200/0] via VTEP 10.1.1.1 VNI 9999 router-mac 00:50:56:96:2a:5a
 B I      192.168.100.1/32 [200/0] via VTEP 10.1.1.1 VNI 9999 router-mac 00:50:56:96:2a:5a
 B I      192.168.100.0/24 [200/0] via VTEP 10.1.1.1 VNI 9999 router-mac 00:50:56:96:2a:5a
 C        192.168.200.0/24 is directly connected, Vlan200

leaf02#

疎通確認

isp02からisp01 Loopback0への疎通が可能になった事を確認します。

isp02#ping 10.11.11.11
PING 10.11.11.11 (10.11.11.11) 72(100) bytes of data.
80 bytes from 10.11.11.11: icmp_seq=1 ttl=62 time=37.6 ms
80 bytes from 10.11.11.11: icmp_seq=2 ttl=62 time=29.0 ms
80 bytes from 10.11.11.11: icmp_seq=3 ttl=62 time=22.0 ms
80 bytes from 10.11.11.11: icmp_seq=4 ttl=62 time=15.1 ms
80 bytes from 10.11.11.11: icmp_seq=5 ttl=62 time=15.1 ms

--- 10.11.11.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 62ms
rtt min/avg/max/mdev = 15.111/23.817/37.637/8.629 ms, pipe 4, ipg/ewma 15.665/30.159 ms
isp02#

動作確認(3) BGPによる接続例

BGPの設定

isp02とleaf02の間でOSPFを設定します。

# ips02
router bgp 65000
   timers bgp 10 30
   neighbor 192.168.200.102 remote-as 65000
   network 10.22.22.22/32

# leaf02
router bgp 65000
   timers bgp 10 30
   vrf GATEWAY
      neighbor 192.168.200.2 remote-as 65000
      address-family ipv4
         neighbor 192.168.200.2 activate

isp01とleaf01の間でBGP neighborが確立された事を確認します。

leaf02#show ip bgp sum vrf GATEWAY
BGP summary information for VRF GATEWAY
Router identifier 192.168.200.102, local AS number 65000
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  192.168.200.2    4 65000              7         7    0    0 00:00:18 Estab   1      1
leaf02#

leaf01, leaf02, isp02は共にiBGP neighborです。そのため、スプリットホライズンの原則が働き、互いに経路を交換しません。スプリットホライズンを打ち消すようroute reflectorを設定します。

router bgp 65000
   neighbor 10.1.1.1 route-reflector-client
   vrf GATEWAY
      neighbor 192.168.200.2 route-reflector-client

leaf01がisp02のプレフィックス(10.22.22.22/32)を学習している事を確認します。

leaf01#show ip route vrf GATEWAY

VRF: GATEWAY
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort is not set

 O        10.11.11.11/32 [110/20] via 192.168.100.1, Vlan100
 B I      10.22.22.22/32 [200/0] via VTEP 10.2.2.2 VNI 9999 router-mac 00:50:56:07:22:5b
 C        192.168.100.0/24 is directly connected, Vlan100
 B I      192.168.200.2/32 [200/0] via VTEP 10.2.2.2 VNI 9999 router-mac 00:50:56:07:22:5b
 B I      192.168.200.0/24 [200/0] via VTEP 10.2.2.2 VNI 9999 router-mac 00:50:56:07:22:5b

leaf01#

isp02がleaf01のプレフィックス(192.168.100.0/24)を学習している事を確認します。

isp02#show ip route

VRF: default
Codes: C - connected, S - static, K - kernel, 
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

Gateway of last resort:
 S        0.0.0.0/0 [1/0] via 192.168.200.254, Vlan200

 B I      10.11.11.11/32 [200/0] via 192.168.200.102, Vlan200
 C        10.22.22.22/32 is directly connected, Loopback0
 B I      192.168.100.1/32 [200/0] via 192.168.200.102, Vlan200
 B I      192.168.100.0/24 [200/0] via 192.168.200.102, Vlan200
 C        192.168.200.0/24 is directly connected, Vlan200

isp02#

疎通確認

isp01とisp02が互いにLoopback同士で疎通可能になった事を確認します。

isp01#ping 10.22.22.22 source Lo0
PING 10.22.22.22 (10.22.22.22) from 10.11.11.11 : 72(100) bytes of data.
80 bytes from 10.22.22.22: icmp_seq=1 ttl=62 time=15.2 ms
80 bytes from 10.22.22.22: icmp_seq=2 ttl=62 time=14.2 ms
80 bytes from 10.22.22.22: icmp_seq=3 ttl=62 time=11.3 ms
80 bytes from 10.22.22.22: icmp_seq=4 ttl=62 time=11.3 ms
80 bytes from 10.22.22.22: icmp_seq=5 ttl=62 time=11.8 ms

--- 10.22.22.22 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 54ms
rtt min/avg/max/mdev = 11.314/12.812/15.279/1.647 ms, pipe 2, ipg/ewma 13.527/13.957 ms
isp01#
タイトルとURLをコピーしました