Clustering provider in Orleans

(Jin Qing's Column, Nov. 3, 2021)

When deployed to a cluster of nodes, Orleans internally implements a protocol to manage it's silos, including discovery, failure and reconfigure, which is called cluster membership management.

Orleans has clustering membership providers for: Azure, SQL server, Zookeeper.

Clustering provider is one of key aspects of silo configuration.

Orleans.Clustering.Kubernetes

OrleansContrib/Orleans.Clustering.Kubernetes is a clustering provider for running Orleans cluster on Kubernetes.

Tell silo to use Kubernetes as the Cluster Membership Provider:

var silo = new SiloBuilder()
        ...
        .UseKubeMembership()
        ...
        .Build();

Interface

UseKubeMembership() instantiates a KubeMembershipTable which implements IMembershipTable.

    public interface IMembershipTable
    {
        Task InitializeMembershipTable(bool tryInitTableVersion);
        Task DeleteMembershipTableEntries(string clusterId);
        Task CleanupDefunctSiloEntries(DateTimeOffset beforeDate);
        Task<MembershipTableData> ReadRow(SiloAddress key);
        Task<MembershipTableData> ReadAll();
        Task<bool> InsertRow(MembershipEntry entry, TableVersion tableVersion);
        Task<bool> UpdateRow(MembershipEntry entry, string etag, TableVersion tableVersion);
        Task UpdateIAmAlive(MembershipEntry entry);
    }

Implement

KubeMembershipTable access Kubernetes API server to read and write silo entry CRD.

• InitializeMembershipTable()

  • TryInitClusterVersion()

    • _kubeClient.GetNamespacedCustomObjectAsync
    • _kubeClient.CreateNamespacedCustomObjectAsync

• DeleteMembershipTableEntries(string clusterId)

  • _kubeClient.DeleteNamespacedCustomObjectAsync

• InsertRow(...)

  • _kubeClient.GetNamespacedCustomObjectAsync
  • _kubeClient.ReplaceNamespacedCustomObjectAsync
  • _kubeClient.CreateNamespacedCustomObjectAsync

• ReadAll()

  • GetClusterVersion()

    • _kubeClient.ListNamespacedCustomObjectAsync

  • GetSilos()

    • _kubeClient.ListNamespacedCustomObjectAsync

• ReadRow(SiloAddress key)

  • _kubeClient.GetNamespacedCustomObjectAsync

• UpdateIAmAlive(MembershipEntry entry)

  • _kubeClient.GetNamespacedCustomObjectAsync
  • _kubeClient.ReplaceNamespacedCustomObjectAsync

• UpdateRow(...)

  • _kubeClient.ReplaceNamespacedCustomObjectAsync

• CleanupDefunctSiloEntries(DateTimeOffset beforeDate)

  • _kubeClient.DeleteNamespacedCustomObjectAsync

The operators to NamespacedCustomObject are:

• Delete
• Get
• Create
• Replace
• List

CRD

Two CRDs, ClusterVersion and Silo are defined in files:

• ClusterVersionCRD.yaml
• SiloEntryCRD.yaml

Custom resource objects are stored in etcd of Kubernetes.