## Scaling Policy

updated: 2020-10-17 18:54

# Scaling Policy

The scaling policy is designed to supplement a cluster scaling request with more detailed arguments based on user-provided rules. This policy type is expected to be applicable on clusters of all profile types.

## Properties

event
  Event that will trigger this policy. Must be one of CLUSTER_SCALE_IN and CLUSTER_SCALE_OUT.
adjustment
  best_effort
    Whether do best effort scaling when new size of cluster will break the size limitation
  cooldown
    Number of seconds to hold the cluster for cool-down before allowing cluster to be resized again.
  min_step
    When adjustment type is set to “CHANGE_IN_PERCENTAGE”, this specifies the cluster size will be decreased by at least this number of nodes.
  number
    A number specifying the amount of adjustment.
  type
    Type of adjustment when scaling is triggered.

## Sample

A typical spec for a scaling policy is shown below:

    # Sample scaling policy that can be attached to a cluster
    type: senlin.policy.scaling
    version: 1.0
    properties:
      event: CLUSTER_SCALE_IN
      adjustment:
        # Adjustment type, valid values include:
        # EXACT_CAPACITY, CHANGE_IN_CAPACITY, CHANGE_IN_PERCENTAGE
        type: CHANGE_IN_CAPACITY

        # A number that will be interpreted based on the type setting.
        number: 1

        # When type is set CHANGE_IN_PERCENTAGE, min_step specifies
        # that the cluster size will be changed by at least the number
        # of nodes specified here.
        min_step: 1

        # When scaling operation will break the size limitation of
        # cluster, whether to do best effort scaling, e.g. decrease
        # cluster size to min_size or increase cluster size to max_size
        # Default False means reject scaling request directly.
        best_effort: True

        # Number of seconds before allowing the cluster to be resized again.
        cooldown: 120

You should pay special attentions to the `event` property, whose valid values include “`CLUSTER_SCALE_IN`” and “`CLUSTER_SCALE_OUT`”. One implication of this design is that you have to attach two policies to the same cluster if you want to control the scaling behavior both when you are expanding the cluster and when you are shrinking it. You can not control the scaling behavior in both directions using the same policy.

Senlin has carefully designed the builtin policy types so that for scaling policies, you can attach more than one instance of the same policy type but you may get an error when you are attempting to attach two policies of another type (say `senlin.policy.deletion`) to the same cluster.

The value of the `event` property indicates when the policy will be checked. A policy with `event` set to “`CLUSTER_SCALE_IN`” will be checked when and only when a corresponding action is triggered on the cluster. A policy with `event` set to “`CLUSTER_SCALE_OUT`” will be checked when and only when a corresponding action is triggered. If the cluster is currently processing a scaling action it will not accept another scaling action until the current action has been processed and cooldown has been observed.

For both types of actions that can triggered the scaling policy, there are always three types of adjustments to choose from as listed below. The type of adjustment determines the interpretation of the `adjustment.number` value.

- `EXACT_CAPACITY`: the value specified for `adjustment.number` means the new capacity of the cluster, so it has to be a non-negative integer.

- `CHANGE_IN_CAPACITY`: the value specified for `adjustment.number` is the number of nodes to be added or removed. This means the value has to be a non-negative number as well.

- `CHANGE_IN_PERCENTAGE`: the value specified for `adjustment.number` will be interpreted as the percentage of capacity changes. This value has to be a non-negative floating-point value.

For example, in the sample spec shown above, when a `CLUSTER_SCALE_IN` request is received, the policy will remove 10% of the total number of nodes from the cluster.

## Dealing With Percentage

As stated above, when `adjustment.type` is set to `CHANGE_IN_PERCENTAGE`, the value of `adjustment.number` can be a floating-point value, interpreted as a percentage of the current node count of the cluster.

In many cases, the result of the calculation may be a floating-point value. For example, if the current capacity of a cluster is 5 and the `adjustment.number` is set to 30%, the compute result will be 1.5. In this situation, the scaling policy rounds the number up to its adjacent integer, i.e. 2. If the `event` property has “`CLUSTER_SCALE_OUT`” as its value, the policy decision is to add 2 nodes to the cluster. If on the other hand the `event` is set to “`CLUSTER_SCALE_IN`”, the policy decision is to remove 2 nodes from the cluster.

There are other corner cases to consider as well. When the compute result is less than 0.1, for example, it becomes a question whether the Senlin engine should add (or remove) nodes. The property `adjustment.min_step` is designed to make this decision. After policy has got the compute result, it will check if it is less than the specified `adjustment.min_step` value and it will use the `adjustment.min_step` value if so.

## Best Effort Scaling

In many auto-scaling usage scenarios, the policy decision may break the size constraints set on the cluster. As an example, a cluster has its `min_size` set to 5, `max_size` set to 10 and its current capacity is 7. If the policy decision is to remove 3 nodes from the cluster, we are in a dilemma. Removing 3 nodes will change the cluster capacity to 4, which is not allowed by the cluster. If we don’t remove 3 nodes, we are not respecting the policy decision.

The `adjustment.best_effort` property is designed to mitigate this situation. When it is set to False, the scaling policy will strictly conform to the rules set. It will reject the scaling request if the computed cluster capacity will break its size constraints. However, if `adjustment.best_effort` is set to True, the scaling policy will strive to compute a sub-optimal number which will not break the cluster’s size constraints. In the above example, this means the policy decision will be “remove 2 nodes from the cluster”. In other words, the policy at least will try partially ful-fill the scaling goal for the sake of respecting the size constraint.

## Cooldown

In real-life cluster deployments, workload pressure fluctuates rapidly. During this minute, it smells like there is a need to add 10 more nodes to handle the bursting workload. During the next minute, it may turn out to be a false alarm, the workload is rapidly decreasing. Since it is very difficult to accurately predict the workload changes, if possible at all, an auto-scaling engine is not supposed to react too prematurely to workload fluctuations.

The `cooldown` property gives you a chance to specify an interval during which the cluster will remain ignorant to scaling requests. Setting a large value to this property will lead to a stable cluster, but the responsiveness to urgent situation is also sacrificed. Setting a small value, on the contrary, can meet the responsiveness requirement, but will also render the cluster into a thrashing state where new nodes are created very frequently only to be removed shortly.

There is never a recommended value that suits all deployments. You will have to try different values in your own environment and tune it for different applications or services.

## Interaction with Other Policies

The scaling policy is only tasked to decide the number of nodes to add or remove. For newly added nodes, you will use other policies to determine where they should be scheduled. For nodes to be deleted, you will use other polices (e.g. the deletion policy) to elect the victim nodes.

The builtin policies were designed carefully so that they can work happily together or by themselves.

updated: 2020-10-17 18:54

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

questions?