--- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.17.3 labels: app.kubernetes.io/instance: kueue app.kubernetes.io/name: kueue app.kubernetes.io/version: v0.12.3 control-plane: controller-manager name: clusterqueues.kueue.x-k8s.io spec: conversion: strategy: Webhook webhook: clientConfig: service: name: kueue-webhook-service namespace: kueue-system path: /convert conversionReviewVersions: - v1 group: kueue.x-k8s.io names: kind: ClusterQueue listKind: ClusterQueueList plural: clusterqueues shortNames: - cq singular: clusterqueue scope: Cluster versions: - additionalPrinterColumns: - description: Cohort that this ClusterQueue belongs to jsonPath: .spec.cohort name: Cohort type: string - description: The queueing strategy used to prioritize workloads jsonPath: .spec.queueingStrategy name: Strategy priority: 1 type: string - description: Number of pending workloads jsonPath: .status.pendingWorkloads name: Pending Workloads type: integer - description: Number of admitted workloads that haven't finished yet jsonPath: .status.admittedWorkloads name: Admitted Workloads priority: 1 type: integer name: v1beta1 schema: openAPIV3Schema: description: ClusterQueue is the Schema for the clusterQueue API. properties: apiVersion: description: |- APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string kind: description: |- Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds type: string metadata: type: object spec: description: ClusterQueueSpec defines the desired state of ClusterQueue properties: admissionChecks: description: |- admissionChecks lists the AdmissionChecks required by this ClusterQueue. Cannot be used along with AdmissionCheckStrategy. items: description: AdmissionCheckReference is the name of an AdmissionCheck. maxLength: 316 type: string type: array admissionChecksStrategy: description: |- admissionCheckStrategy defines a list of strategies to determine which ResourceFlavors require AdmissionChecks. This property cannot be used in conjunction with the 'admissionChecks' property. properties: admissionChecks: description: admissionChecks is a list of strategies for AdmissionChecks items: description: AdmissionCheckStrategyRule defines rules for a single AdmissionCheck properties: name: description: name is an AdmissionCheck's name. maxLength: 316 type: string onFlavors: description: |- onFlavors is a list of ResourceFlavors' names that this AdmissionCheck should run for. If empty, the AdmissionCheck will run for all workloads submitted to the ClusterQueue. items: description: ResourceFlavorReference is the name of the ResourceFlavor. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string type: array required: - name type: object type: array type: object admissionScope: description: admissionScope indicates whether ClusterQueue uses the Admission Fair Sharing properties: admissionMode: description: |- AdmissionMode indicates which mode for AdmissionFairSharing should be used in the AdmissionScope. Possible values are: - UsageBasedAdmissionFairSharing - NoAdmissionFairSharing type: string required: - admissionMode type: object cohort: description: |- cohort that this ClusterQueue belongs to. CQs that belong to the same cohort can borrow unused resources from each other. A CQ can be a member of a single borrowing cohort. A workload submitted to a queue referencing this CQ can borrow quota from any CQ in the cohort. Only quota for the [resource, flavor] pairs listed in the CQ can be borrowed. If empty, this ClusterQueue cannot borrow from any other ClusterQueue and vice versa. A cohort is a name that links CQs together, but it doesn't reference any object. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string fairSharing: description: |- fairSharing defines the properties of the ClusterQueue when participating in FairSharing. The values are only relevant if FairSharing is enabled in the Kueue configuration. properties: weight: anyOf: - type: integer - type: string default: 1 description: |- weight gives a comparative advantage to this ClusterQueue or Cohort when competing for unused resources in the Cohort. The share is based on the dominant resource usage above nominal quotas for each resource, divided by the weight. Admission prioritizes scheduling workloads from ClusterQueues and Cohorts with the lowest share and preempting workloads from the ClusterQueues and Cohorts with the highest share. A zero weight implies infinite share value, meaning that this Node will always be at disadvantage against other ClusterQueues and Cohorts. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object flavorFungibility: default: {} description: |- flavorFungibility defines whether a workload should try the next flavor before borrowing or preempting in the flavor being evaluated. properties: whenCanBorrow: default: Borrow description: |- whenCanBorrow determines whether a workload should try the next flavor before borrowing in current flavor. The possible values are: - `Borrow` (default): allocate in current flavor if borrowing is possible. - `TryNextFlavor`: try next flavor even if the current flavor has enough resources to borrow. enum: - Borrow - TryNextFlavor type: string whenCanPreempt: default: TryNextFlavor description: |- whenCanPreempt determines whether a workload should try the next flavor before borrowing in current flavor. The possible values are: - `Preempt`: allocate in current flavor if it's possible to preempt some workloads. - `TryNextFlavor` (default): try next flavor even if there are enough candidates for preemption in the current flavor. enum: - Preempt - TryNextFlavor type: string type: object namespaceSelector: description: |- namespaceSelector defines which namespaces are allowed to submit workloads to this clusterQueue. Beyond this basic support for policy, a policy agent like Gatekeeper should be used to enforce more advanced policies. Defaults to null which is a nothing selector (no namespaces eligible). If set to an empty selector `{}`, then all namespaces are eligible. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: |- A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: |- operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: |- values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array x-kubernetes-list-type: atomic required: - key - operator type: object type: array x-kubernetes-list-type: atomic matchLabels: additionalProperties: type: string description: |- matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic preemption: default: {} description: |- ClusterQueuePreemption contains policies to preempt Workloads from this ClusterQueue or the ClusterQueue's cohort. Preemption may be configured to work in the following scenarios: - When a Workload fits within the nominal quota of the ClusterQueue, but the quota is currently borrowed by other ClusterQueues in the cohort. We preempt workloads in other ClusterQueues to allow this ClusterQueue to reclaim its nominal quota. Configured using reclaimWithinCohort. - When a Workload doesn't fit within the nominal quota of the ClusterQueue and there are admitted Workloads in the ClusterQueue with lower priority. Configured using withinClusterQueue. - When a Workload may fit while both borrowing and preempting low priority workloads in the Cohort. Configured using borrowWithinCohort. - When FairSharing is enabled, to maintain fair distribution of unused resources. See FairSharing documentation. The preemption algorithm tries to find a minimal set of Workloads to preempt to accomomdate the pending Workload, preempting Workloads with lower priority first. properties: borrowWithinCohort: default: {} description: |- BorrowWithinCohort contains configuration which allows to preempt workloads within cohort while borrowing. It only works with Classical Preemption, __not__ with Fair Sharing. properties: maxPriorityThreshold: description: |- maxPriorityThreshold allows to restrict the set of workloads which might be preempted by a borrowing workload, to only workloads with priority less than or equal to the specified threshold priority. When the threshold is not specified, then any workload satisfying the policy can be preempted by the borrowing workload. format: int32 type: integer policy: default: Never description: |- policy determines the policy for preemption to reclaim quota within cohort while borrowing. Possible values are: - `Never` (default): do not allow for preemption, in other ClusterQueues within the cohort, for a borrowing workload. - `LowerPriority`: allow preemption, in other ClusterQueues within the cohort, for a borrowing workload, but only if the preempted workloads are of lower priority. enum: - Never - LowerPriority type: string type: object reclaimWithinCohort: default: Never description: |- reclaimWithinCohort determines whether a pending Workload can preempt Workloads from other ClusterQueues in the cohort that are using more than their nominal quota. The possible values are: - `Never` (default): do not preempt Workloads in the cohort. - `LowerPriority`: **Classic Preemption** if the pending Workload fits within the nominal quota of its ClusterQueue, only preempt Workloads in the cohort that have lower priority than the pending Workload. **Fair Sharing** only preempt Workloads in the cohort that have lower priority than the pending Workload and that satisfy the Fair Sharing preemptionStategies. - `Any`: **Classic Preemption** if the pending Workload fits within the nominal quota of its ClusterQueue, preempt any Workload in the cohort, irrespective of priority. **Fair Sharing** preempt Workloads in the cohort that satisfy the Fair Sharing preemptionStrategies. enum: - Never - LowerPriority - Any type: string withinClusterQueue: default: Never description: |- withinClusterQueue determines whether a pending Workload that doesn't fit within the nominal quota for its ClusterQueue, can preempt active Workloads in the ClusterQueue. The possible values are: - `Never` (default): do not preempt Workloads in the ClusterQueue. - `LowerPriority`: only preempt Workloads in the ClusterQueue that have lower priority than the pending Workload. - `LowerOrNewerEqualPriority`: only preempt Workloads in the ClusterQueue that either have a lower priority than the pending workload or equal priority and are newer than the pending workload. enum: - Never - LowerPriority - LowerOrNewerEqualPriority type: string type: object x-kubernetes-validations: - message: reclaimWithinCohort=Never and borrowWithinCohort.Policy!=Never rule: '!(self.reclaimWithinCohort == ''Never'' && has(self.borrowWithinCohort) && self.borrowWithinCohort.policy != ''Never'')' queueingStrategy: default: BestEffortFIFO description: |- QueueingStrategy indicates the queueing strategy of the workloads across the queues in this ClusterQueue. Current Supported Strategies: - StrictFIFO: workloads are ordered strictly by creation time. Older workloads that can't be admitted will block admitting newer workloads even if they fit available quota. - BestEffortFIFO: workloads are ordered by creation time, however older workloads that can't be admitted will not block admitting newer workloads that fit existing quota. enum: - StrictFIFO - BestEffortFIFO type: string resourceGroups: description: |- resourceGroups describes groups of resources. Each resource group defines the list of resources and a list of flavors that provide quotas for these resources. Each resource and each flavor can only form part of one resource group. resourceGroups can be up to 16. items: properties: coveredResources: description: |- coveredResources is the list of resources covered by the flavors in this group. Examples: cpu, memory, vendor.com/gpu. The list cannot be empty and it can contain up to 16 resources. items: description: ResourceName is the name identifying various resources in a ResourceList. type: string maxItems: 16 minItems: 1 type: array flavors: description: |- flavors is the list of flavors that provide the resources of this group. Typically, different flavors represent different hardware models (e.g., gpu models, cpu architectures) or pricing models (on-demand vs spot cpus). Each flavor MUST list all the resources listed for this group in the same order as the .resources field. The list cannot be empty and it can contain up to 16 flavors. items: properties: name: description: |- name of this flavor. The name should match the .metadata.name of a ResourceFlavor. If a matching ResourceFlavor does not exist, the ClusterQueue will have an Active condition set to False. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string resources: description: |- resources is the list of quotas for this flavor per resource. There could be up to 16 resources. items: properties: borrowingLimit: anyOf: - type: integer - type: string description: |- borrowingLimit is the maximum amount of quota for the [flavor, resource] combination that this ClusterQueue is allowed to borrow from the unused quota of other ClusterQueues in the same cohort. In total, at a given time, Workloads in a ClusterQueue can consume a quantity of quota equal to nominalQuota+borrowingLimit, assuming the other ClusterQueues in the cohort have enough unused quota. If null, it means that there is no borrowing limit. If not null, it must be non-negative. borrowingLimit must be null if spec.cohort is empty. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true lendingLimit: anyOf: - type: integer - type: string description: |- lendingLimit is the maximum amount of unused quota for the [flavor, resource] combination that this ClusterQueue can lend to other ClusterQueues in the same cohort. In total, at a given time, ClusterQueue reserves for its exclusive use a quantity of quota equals to nominalQuota - lendingLimit. If null, it means that there is no lending limit, meaning that all the nominalQuota can be borrowed by other clusterQueues in the cohort. If not null, it must be non-negative. lendingLimit must be null if spec.cohort is empty. This field is in beta stage and is enabled by default. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true name: description: name of this resource. type: string nominalQuota: anyOf: - type: integer - type: string description: |- nominalQuota is the quantity of this resource that is available for Workloads admitted by this ClusterQueue at a point in time. The nominalQuota must be non-negative. nominalQuota should represent the resources in the cluster available for running jobs (after discounting resources consumed by system components and pods not managed by kueue). In an autoscaled cluster, nominalQuota should account for resources that can be provided by a component such as Kubernetes cluster-autoscaler. If the ClusterQueue belongs to a cohort, the sum of the quotas for each (flavor, resource) combination defines the maximum quantity that can be allocated by a ClusterQueue in the cohort. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true required: - name - nominalQuota type: object maxItems: 16 minItems: 1 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name - resources type: object maxItems: 16 minItems: 1 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - coveredResources - flavors type: object x-kubernetes-validations: - message: flavors must have the same number of resources as the coveredResources rule: self.flavors.all(x, size(x.resources) == size(self.coveredResources)) maxItems: 16 type: array x-kubernetes-list-type: atomic stopPolicy: default: None description: |- stopPolicy - if set to a value different from None, the ClusterQueue is considered Inactive, no new reservation being made. Depending on its value, its associated workloads will: - None - Workloads are admitted - HoldAndDrain - Admitted workloads are evicted and Reserving workloads will cancel the reservation. - Hold - Admitted workloads will run to completion and Reserving workloads will cancel the reservation. enum: - None - Hold - HoldAndDrain type: string type: object x-kubernetes-validations: - message: borrowingLimit must be nil when cohort is empty rule: '!has(self.cohort) && has(self.resourceGroups) ? self.resourceGroups.all(rg, rg.flavors.all(f, f.resources.all(r, !has(r.borrowingLimit)))) : true' status: description: ClusterQueueStatus defines the observed state of ClusterQueue properties: admittedWorkloads: description: |- admittedWorkloads is the number of workloads currently admitted to this clusterQueue and haven't finished yet. format: int32 type: integer conditions: description: |- conditions hold the latest available observations of the ClusterQueue current state. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: lastTransitionTime: description: |- lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. format: date-time type: string message: description: |- message is a human readable message indicating details about the transition. This may be an empty string. maxLength: 32768 type: string observedGeneration: description: |- observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. format: int64 minimum: 0 type: integer reason: description: |- reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. maxLength: 1024 minLength: 1 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ type: string status: description: status of the condition, one of True, False, Unknown. enum: - "True" - "False" - Unknown type: string type: description: type of condition in CamelCase or in foo.example.com/CamelCase. maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string required: - lastTransitionTime - message - reason - status - type type: object type: array x-kubernetes-list-map-keys: - type x-kubernetes-list-type: map fairSharing: description: |- fairSharing contains the current state for this ClusterQueue when participating in Fair Sharing. This is recorded only when Fair Sharing is enabled in the Kueue configuration. properties: admissionFairSharingStatus: description: admissionFairSharingStatus represents information relevant to the Admission Fair Sharing properties: consumedResources: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: |- ConsumedResources represents the aggregated usage of resources over time, with decaying function applied. The value is populated if usage consumption functionality is enabled in Kueue config. type: object lastUpdate: description: LastUpdate is the time when share and consumed resources were updated. format: date-time type: string required: - consumedResources - lastUpdate type: object weightedShare: description: |- WeightedShare represents the maximum of the ratios of usage above nominal quota to the lendable resources in the Cohort, among all the resources provided by the Node, and divided by the weight. If zero, it means that the usage of the Node is below the nominal quota. If the Node has a weight of zero and is borrowing, this will return 9223372036854775807, the maximum possible share value. format: int64 type: integer required: - weightedShare type: object flavorsReservation: description: |- flavorsReservation are the reserved quotas, by flavor, currently in use by the workloads assigned to this ClusterQueue. items: properties: name: description: name of the flavor. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string resources: description: resources lists the quota usage for the resources in this flavor. items: properties: borrowed: anyOf: - type: integer - type: string description: |- Borrowed is quantity of quota that is borrowed from the cohort. In other words, it's the used quota that is over the nominalQuota. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true name: description: name of the resource type: string total: anyOf: - type: integer - type: string description: |- total is the total quantity of used quota, including the amount borrowed from the cohort. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true required: - name type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name - resources type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map flavorsUsage: description: |- flavorsUsage are the used quotas, by flavor, currently in use by the workloads admitted in this ClusterQueue. items: properties: name: description: name of the flavor. maxLength: 253 pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ type: string resources: description: resources lists the quota usage for the resources in this flavor. items: properties: borrowed: anyOf: - type: integer - type: string description: |- Borrowed is quantity of quota that is borrowed from the cohort. In other words, it's the used quota that is over the nominalQuota. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true name: description: name of the resource type: string total: anyOf: - type: integer - type: string description: |- total is the total quantity of used quota, including the amount borrowed from the cohort. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true required: - name type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map required: - name - resources type: object maxItems: 16 type: array x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map pendingWorkloads: description: |- pendingWorkloads is the number of workloads currently waiting to be admitted to this clusterQueue. format: int32 type: integer pendingWorkloadsStatus: description: |- PendingWorkloadsStatus contains the information exposed about the current status of the pending workloads in the cluster queue. Deprecated: This field will be removed on v1beta2, use VisibilityOnDemand (https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand/) instead. properties: clusterQueuePendingWorkload: description: Head contains the list of top pending workloads. items: description: |- ClusterQueuePendingWorkload contains the information identifying a pending workload in the cluster queue. properties: name: description: Name indicates the name of the pending workload. type: string namespace: description: Namespace indicates the name of the pending workload. type: string required: - name - namespace type: object type: array x-kubernetes-list-type: atomic lastChangeTime: description: LastChangeTime indicates the time of the last change of the structure. format: date-time type: string required: - lastChangeTime type: object reservingWorkloads: description: |- reservingWorkloads is the number of workloads currently reserving quota in this clusterQueue. format: int32 type: integer type: object type: object served: true storage: true subresources: status: {}