Skip to main content

What is a Zone

A Zone is used to separate different teams or products that share the same Kubernetes cluster. Improved security through access permissions, policies, network rules, and dedicated compute resources results in far better isolation than using Namespaces alone. The main reasons for using Zones are security and cost savings.

A Zone contains Namespaces. A Zone can create Kubernetes Node Pools (in AWS, NodeGroups). Pods assigned to a Namespace that belongs to a Zone automatically receive the appropriate NodeSelectors and are scheduled on the correct Node Pool. The PSA of the Namespace is controlled by the Zone, preventing any privileged containers from starting. The PSA enforcement level is configurable at the cluster level. Since a Zone can contain multiple Namespaces and Node Pools, it is also possible to specify which Node Pools are used by each Namespace in a Zone. Per-Pod Affinity is also supported. This is convenient for users, as the required affinity rules are applied automatically unless the user wants to control them manually. In the latter case, validations are in place to enforce the rules.

If the granularNamespaceNetworkPolicy is false(default) then the Zone allows network traffic between Namespaces that belong to the same Zone. When Namespaces in different Zones need to communicate, they must always allow incoming connections using standard Kubernetes NetworkPolicy objects. Only NetworkPolicy ingress rules need to be added; egress is open by default.

When Ingress objects are created, the necessary NetworkPolicy for the load balancer to access the Service Pods is created automatically.

Each Zone also creates an ArgoCDProject with permissions to deploy resources to the Zone's Namespaces, enabling the use of "app-of-apps" per team while keeping them within the designated Zone.

Zones can also assign Contributor groups to restrict access to only the Namespaces within the Zone. This access is granted in both ArgoCD and the Kubernetes API.

A Demo video of Zone can be found here https://youtu.be/D1Eg0rE3nSI.

Assigning Permissions​

There are two configurations for roleMapping that determine the permissions of user groups: global permissions, assignable through the "platform-api" Helm chart values, and per-Zone permissions, assignable through the Zone object spec. Globally assigned roles apply across all Zones and their Namespaces in the Kubernetes cluster. Per-Zone permissions apply only to that Zone's Namespaces.

The permissions a role has via the Kubernetes API (kubectl) are nearly identical to the permissions granted in the Zone's ArgoCDProject. Due to technical limitations, these permissions cannot be 100% equal.

Detailed Role Descriptions​

Available roles and their purpose.

Observer is intended for users who need read-only access to applications without the ability to make changes. It can read objects but not modify them. Can read cluster-scoped objects: namespaces, nodes, persistentvolumes, storageclasses, ingressclasses, ingressclassparams.elbv2.k8s.aws, zones, nodegroups.eks.aws.upbound.io. It can read all namespace-scoped objects in the Zone's Namespaces.

Contributor is intended for users who need to deploy and manage applications within the Zone boundary. It can create new Namespaces in Zones (through ArgoCDApplications, but not kubectl; the Namespace is assigned to the Zone whose ArgoCDProject it was created in). It can read the same cluster-scoped objects as Observer. Can read, create, edit, and delete all namespace-scoped objects in the Zone's Namespaces, including ArgoCDApplication objects, but only within the Zone's ArgoCDProject. Can configure ArgoCDApplications and repositories.

Maintainer is intended for configuring Zone objects and enabling Contributors to deploy applications within the Zone boundary. Maintainers can also perform all Contributor duties. A Maintainer is less powerful than a cluster administrator and has no permissions to manage the "infralib" Zone or its Namespaces. It can read, create, edit, and delete Zones and Namespaces. Can assign user groups to Contributor permissions in a Zone using the spec.roleMapping field in the Zone object. Has all permissions of Contributor. Due to the presence of spec.clusterPermissions in the Zone object, a Maintainer is able to escalate privileges to cluster administrator.

Future state: Currently the capabilities and existence of these roles are provided by the platform-api. In the future it should be possible to tune the permissions and the roles that exist. This is why the roleMapping uses roleRef as a text field.

Infralib Configuration Example for Global/Cluster-wide roleMappings​

steps:
  ...
  - name: apps
    type: argocd-apps
    modules:
    ...
    - name: platform-apis
      source: platform-apis
      inputs:
        platform-apis:
          zone:
            environmentConfig:
              roleMapping:
                - roleRef: maintainer
                  groups: ["maintainer-group"]
                - roleRef: observer
                  groups: ["observer-group"]

Infralib Configuration Example for Per-Zone roleMappings​

apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zone
spec:
  roleMapping:
    - groups:
        - contributor-group
      roleRef: contributor
    - groups:
        - observer-group
      roleRef: observer

Infralib Configuration Example for Tuning ArgoCDProject Permissions​

It is possible to change what resources are whitelisted or blacklisted in the Zone ArgoCDProject or the project "zone" using Helm values in the platform-apis.

        platform-apis:
          zone:
            environmentConfig:
              appProject:
                namespaceResourceBlacklist: #Affects the namespace resource blacklist of the Zone ArgoCDProject. Example contains default values.
                - group: '*.m.upbound.io'
                  kind: '*'
                namespaceResourceWhitelist: []
          clusterResourceWhitelist: #Affects the cluster resource whitelist of the "zone" project. Example contains default values.
          - group: tenancy.entigo.com
            kind: Zone
          - group: eks.aws.upbound.io
            kind: NodeGroup

Example of a Zone​

This is an example of how to create a zone. This will create two Node Groups in AWS: "default" and "myspot".

It is a good practice to manage zones using a GitOps methodology, similar to how applications are deployed. For better organization, consider creating a dedicated Git repository and ArgoCDProject specifically for zones.

Use a Git repository with the zone manifest, or create the zone using kubectl.

# Example zone
apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zone
  labels:
    tenancy.entigo.com/default-zone: "true"
spec:
  roleMapping:
    - groups:
        - 123456789-1234-1234-1234-123456789
      roleRef: contributor
  pools:
    - name: default
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 2
    - name: myspot
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: SPOT
        - key: min-size
          value: 2
        - key: max-size
          value: 2

Example for Zonal NodeGroups​

In this example we want to create NodeGroups per zone and assign a StatefulSet to be deployed across multiple zones.

# Example zone
apiVersion: tenancy.entigo.com/v1alpha1
kind: Zone
metadata:
  name: example-zonal
spec:
  pools:
    - name: default-a
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 1
        - key: zone
          values:
            - eu-north-1a
    - name: default-b
      requirements:
        - key: instance-type
          values:
            - t3.large
        - key: capacity-type
          value: ON_DEMAND
        - key: min-size
          value: 1
        - key: max-size
          value: 1
        - key: zone
          values:
            - eu-north-1b

Then add a podAntiAffinity rule for the Pods with topology topology.kubernetes.io/zone.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: example
  name: example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - example
              topologyKey: topology.kubernetes.io/zone
      containers:
        - image: nginxinc/nginx-unprivileged
          name: nginx

Create a Zone Using ArgoCD​

Please watch the demo video https://youtu.be/D1Eg0rE3nSI

The ArgoCD App of Apps pattern​

A Namespace for ArgoCD Application and ApplicationSet objects is automatically created when a Zone is created. The naming convention is <zone-name>-apps. The purpose of this Namespace is to make the app-of-apps pattern easy to use with Zones. Otherwise, this Namespace would have to be created manually. Users of a Zone are encouraged to create their ArgoCD Applications in this Namespace and have those Applications create the subsequent Namespaces for application resources.

Creating Applications in an ArgoCD Project that belongs to the Zone will automatically assign the same Zone to any Namespaces that are created. The ArgoCD Application destination namespace is tracked for this purpose.

This apps Namespace is managed by the Zone itself and cannot be moved to another Zone. It is also deleted when a Zone is deleted, which is why no other resources are permitted inside it.

To disable the creation of this Namespace, set createAppsNamespace to false in the platform-api Helm values file.

This pattern is also show in the demo video https://www.youtube.com/watch?v=D1Eg0rE3nSI&t=180

Assign a Namespace to a Zone​

$ kubectl create ns example-ns

Since the "example-zone" zone has the label tenancy.entigo.com/default-zone with value true, the Namespace is automatically assigned to that Zone. If no Zone has this label, the first available Zone is chosen.

To change the Zone of the Namespace example-ns to the Zone another-zone, update the tenancy.entigo.com/zone label on the Namespace.

$ kubectl label ns example-ns tenancy.entigo.com/zone=another-zone --overwrite

It is possible to assign a Zone to a Namespace immediately by defining the labels.

apiVersion: v1
kind: Namespace
metadata:
  name: a2
  labels:
    tenancy.entigo.com/zone: example-zone
    tenancy.entigo.com/pool: myspot

To specify the Pool to use for this Namespace, set the tenancy.entigo.com/pool label. This will schedule Pods in this Namespace onto that pool of nodes.

After the Namespace is created, the permissions are also applied to the Zone's ArgoCDProject, allowing these Namespaces to be used from ArgoCD.

If the user does not want to create Namespaces using kubectl, they can also create a new Application in ArgoCD. When this Application belongs to a Zone's project, the destination Namespace is automatically created and assigned to the same Zone.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: a3
  namespace: a-apps
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: a3
  project: example-zone
  source:
    ...

If the "a3" Namespace did not exist, it will be created and assigned to the example-zone Zone.