Ceph @ BIT

Stefan Kooman, stefan@bit.nl, @basseroet BIT

Agenda

  • From Storage vendor to Open Source Storage
  • Requirements for new storage solution
  • Ceph to the rescue!
  • What will we use Ceph for?
  • Timeline Ceph @ BIT
  • Design production cluster
  • Build the cluster
  • Development challenges

From Storage vendor to Open Source Storage

Reasons to migrate

  • Performance (lack thereof)
  • Does not scale horizontally
  • Lots of critical bugs
  • Lots of maintenance
  • Not happy with support

Looking for alternatives

Laws of the Widos 1/2

Wido Potters (Manager CCS @ BIT):

  • 1) The first 6 months you will be in love with your new storage from vendor X
  • 2) After 6 months cracks arise in the relationship
  • 3) After one year you are totally fed up with your storage solution

Functional Requirements

  • Redundant Block Storage (iSCSI)
  • Redundant File Storage (nfs)
  • Redundant Object Storage (S3)
  • Cluster remain functional in case of a Data center failure
  • low maintenance
  • stable platform
  • horizontally scalable (makes sense for economics too)

Considerations

  • Eat your own dog food
  • No difference between BIT / Customers

Ceph to the rescue!

    Provides us with:
  • Reduindant Block Storage: RBD
  • Redundant File Storage: Cephfs
  • Redundant Object Storage: RGW

Hey, we need support for legacy applications

  • NFS Ganesha on top of Cephfs
  • SuSE iSCSI Gateway (lrbd) on top of Ceph rbd
  • or
  • Ceph iSCSI Gateway on top of Ceph rbd

Ceph @ BIT Timeline

Ceph Test cluster

Ceph Test cluster

(I made this mess, don't blame DC management)

Ceph Test Cluster

  • Useful to test switches
  • Get familiar with Ceph
  • Test Ceph upgrades
  • Test failures
  • Plan for production

Design Production Cluster

  • IPv6 Only (Ceph not dual stack)
  • Access to Ceph storage through BIT Access Network
  • Triple replicas: Datacenter BIT-1, BIT-2A, BIT-2C
  • BGP Routed
  • BGP ECMP across links (easy scaling bandwith)
  • BGP EVPN (VXlan)
  • Redundant paths between DC's (two separate fiber rings)
  • No better kill than overkill! :p

Ceph Production Cluster

Build Production Cluster

Laws of the Widos 2/2

    Wido den Hollander (42on):

  • Ceph IPv6
  • one (public) network
  • one interface
  • one IPv6 address

OOB Management

    Wish for out of band management
  • Linux network namespaces
  • Complete isolation between network stacks

mgmt namespace


/sbin/ip netns add mgmt
/sbin/ip link set $INTERFACE netns mgmt
/sbin/ip netns exec mgmt /sbin/ip link set lo up
/sbin/ip netns exec mgmt /sbin/ip link set $INTERFACE up
/sbin/ip netns exec mgmt /sbin/ip -6 addr add 2001:deadbeef/64 dev $INTERFACE
/sbin/ip netns exec mgmt /sbin/ip -6 route del default
/sbin/ip netns exec mgmt /sbin/ip -6 route add default via $GATEWAY6

Why a seperate management network?

  • Troubleshoot Ceph / network issues through ssh
  • Always Collect metrics
  • ... to ease future changes
  • DPDK in Ceph
  • Make sure daemons live in right namespace

  • sshd
  • rsyslogd
  • bit-monitoring
  • telegraf
  • snappy
  • more to come?
  • Infra as code

    • GIT (instead of SVN)
    • Ansible (instead of cfengine2)
    • GITLAB (workflow)

    Switch infra

    • Storage network similar to access network
    • Provisioned in same way (GIT, Ansible, GITLAB)
    • OOB for switches (Arista) implemented in same way (namespaces)
    • IPv6 only OOB

    Monitoring / Metrics

    • Near real-time info
    • Alerting

    Monitoring / Metrics

    • Icinga (alerting)
    • InfluxDB (storing metrics)
    • Grafana (viewing metrics)
    • Telegraf (collecting metrics)
    • Snappy (collecting Ceph metrics)
    • AMP (continuously perform active network measurements between a mesh of Ceph nodes)

    Ceph tools

    • ceph -s
    • ceph osd tree

    ceph -s

    
      cluster:
        id:     7d44bbde-2442-59dc-8c71-e7dad785c99b
        health: HEALTH_WARN
                1 datacenter (21 osds) down
                21 osds down
                3 hosts (21 osds) down
                1 rack (21 osds) down
                Degraded data redundancy: 19469/56142 objects degraded (34.678%), 2048 pgs unclean, 2048 pgs degraded, 2048 pgs undersized
                2/5 mons down, quorum mon5,mon3,mon4
    
    services:
        mon: 5 daemons, quorum mon5,mon3,mon4, out of quorum: mon1, mon2
        mgr: mon4(active), standbys: mon3, mon5
        osd: 63 osds: 41 up, 62 in
    

    ceph osd tree

    
    ceph osd tree
    ID  CLASS WEIGHT   TYPE NAME              STATUS REWEIGHT PRI-AFF 
    -21       69.29997 region BIT-Ede                                 
    -23       23.09999     datacenter BIT-1                           
    -26       23.09999         rack rack1                             
     -3        7.70000             host osd1                          
      0   ssd  1.09999                 osd.0    down  1.00000 1.00000 
      1   ssd  1.09999                 osd.1    down  1.00000 1.00000 
      2   ssd  1.09999                 osd.2    down  1.00000 1.00000 
      3   ssd  1.09999                 osd.3    down  1.00000 1.00000 
      4   ssd  1.09999                 osd.4    down  1.00000 1.00000 
      5   ssd  1.09999                 osd.5    down  1.00000 1.00000 
      6   ssd  1.09999                 osd.6    down  1.00000 1.00000 
     -2        7.70000             host osd2                          
      7   ssd  1.09999                 osd.7    down  1.00000 1.00000 
      8   ssd  1.09999                 osd.8    down  1.00000 1.00000 
      9   ssd  1.09999                 osd.9    down  1.00000 1.00000 
     10   ssd  1.09999                 osd.10   down  1.00000 1.00000 
     11   ssd  1.09999                 osd.11   down  1.00000 1.00000 
     12   ssd  1.09999                 osd.12   down  1.00000 1.00000 
     13   ssd  1.09999                 osd.13   down  1.00000 1.00000 
     -5        7.70000             host osd3                          
     14   ssd  1.09999                 osd.14   down  1.00000 1.00000 
     15   ssd  1.09999                 osd.15   down  1.00000 1.00000 
     16   ssd  1.09999                 osd.16   down  1.00000 1.00000 
     17   ssd  1.09999                 osd.17   down  1.00000 1.00000 
     18   ssd  1.09999                 osd.18   down  1.00000 1.00000 
     19   ssd  1.09999                 osd.19   down  1.00000 1.00000 
     20   ssd  1.09999                 osd.20   down  1.00000 1.00000 
    -24       23.09999     datacenter BIT-2A                          
    -27       23.09999         rack rack2                             
     -6        7.70000             host osd4                          
     21   ssd  1.09999                 osd.21     up  1.00000 1.00000 
     22   ssd  1.09999                 osd.22     up  1.00000 1.00000 
     23   ssd  1.09999                 osd.23     up  1.00000 1.00000 
     24   ssd  1.09999                 osd.24     up  1.00000 1.00000 
     25   ssd  1.09999                 osd.25     up  1.00000 1.00000 
     26   ssd  1.09999                 osd.26     up  1.00000 1.00000 
     27   ssd  1.09999                 osd.27     up  1.00000 1.00000 
     -4        7.70000             host osd5                          
     28   ssd  1.09999                 osd.28     up  1.00000 1.00000 
     29   ssd  1.09999                 osd.29     up  1.00000 1.00000 
     30   ssd  1.09999                 osd.30     up  1.00000 1.00000 
     31   ssd  1.09999                 osd.31     up  1.00000 1.00000 
     32   ssd  1.09999                 osd.32     up  1.00000 1.00000 
     33   ssd  1.09999                 osd.33     up  1.00000 1.00000 
     34   ssd  1.09999                 osd.34     up  1.00000 1.00000 
     -7        7.70000             host osd6                          
     35   ssd  1.09999                 osd.35     up  1.00000 1.00000 
     36   ssd  1.09999                 osd.36     up  1.00000 1.00000 
     37   ssd  1.09999                 osd.37     up  1.00000 1.00000 
     38   ssd  1.09999                 osd.38     up  1.00000 1.00000 
     39   ssd  1.09999                 osd.39     up  1.00000 1.00000 
     40   ssd  1.09999                 osd.40     up  1.00000 1.00000 
     41   ssd  1.09999                 osd.41     up  1.00000 1.00000 
    -22       23.09999     datacenter BIT-2C                          
    -25       23.09999         rack rack3                             
     -8        7.70000             host osd7                          
     42   ssd  1.09999                 osd.42     up  1.00000 1.00000 
     43   ssd  1.09999                 osd.43     up  1.00000 1.00000 
     44   ssd  1.09999                 osd.44     up  1.00000 1.00000 
     45   ssd  1.09999                 osd.45     up  1.00000 1.00000 
     46   ssd  1.09999                 osd.46     up  1.00000 1.00000 
     47   ssd  1.09999                 osd.47     up  1.00000 1.00000 
     48   ssd  1.09999                 osd.48     up  1.00000 1.00000 
     -9        7.70000             host osd8                          
     49   ssd  1.09999                 osd.49     up  1.00000 1.00000 
     50   ssd  1.09999                 osd.50     up  1.00000 1.00000 
     51   ssd  1.09999                 osd.51     up  1.00000 1.00000 
     52   ssd  1.09999                 osd.52     up  1.00000 1.00000 
     53   ssd  1.09999                 osd.53     up  1.00000 1.00000 
     54   ssd  1.09999                 osd.54     up  1.00000 1.00000 
     55   ssd  1.09999                 osd.55     up  1.00000 1.00000 
    -10        7.70000             host osd9                          
     56   ssd  1.09999                 osd.56     up  1.00000 1.00000 
     57   ssd  1.09999                 osd.57     up  1.00000 1.00000 
     58   ssd  1.09999                 osd.58     up  1.00000 1.00000 
     59   ssd  1.09999                 osd.59     up  1.00000 1.00000 
     60   ssd  1.09999                 osd.60     up  1.00000 1.00000 
     61   ssd  1.09999                 osd.61     up  1.00000 1.00000 
     62   ssd  1.09999                 osd.62     up  1.00000 1.00000 
     -1              0 root default
    

    Ceph Performance Tuning

  • CPU governor: Powersave
  • /mnt/perfdisk1# ioping -D .
    
    --- . (xfs /dev/rbd0) ioping statistics ---
    7.20 k requests completed in 2.00 hour, 1.51 k iops, 5.89 MiB/s
    min/avg/max/mdev = 378 us / 663 us / 1.15 ms / 90 us
    

    Ceph Performance Tuning

  • CPU governor: Performance
  • /mnt/perfdisk1# ioping -D .
    4 KiB from . (xfs /dev/rbd0): request=1 time=337 us
    4 KiB from . (xfs /dev/rbd0): request=2 time=566 us
    4 KiB from . (xfs /dev/rbd0): request=3 time=602 us
    ...
    --- . (xfs /dev/rbd0) ioping statistics ---
    7.20 k requests completed in 2.00 hour, 2.08 k iops, 8.12 MiB/s
    min/avg/max/mdev = 214 us / 481 us / 937 us / 128 us
    
    182 us difference (~ 38 % increase in latency, 38% decrease in throughput) At the cost of 15% increase in power usage (performance mode)

    Cluster on the move

    All infra in one rack (easy installation / configuration)

    Cluster on the move

    All infra in one rack (easy configuration)

    Challenges during development

    • Bootstrap cluster with Ansible (mon <-> mgr)
    • Namespace challenge (Ceph commands in default namespace)
    • Need to clear IPv6 neighbour cache after port flap
    • IPv6 only bug in Ansible (#26740)
    • Need to reboot all mons after upgrade before Ceph started working again (Luminous RC issue)
    • Benchmarking is hard

    Almost production ready

    • Change to BlueStore
    • Final benchmarking
    • Metrics/dashboard/alerting
    • Final Domain Failure tests under load
    • Final fiber ring infra / patching