Ensuring One-Time Execution of Deployment Tasks in Multi-Instance Environments AWS Code Pipeline

Overview

In an environment where multiple instances of an application are deployed, ensuring that specific deployment tasks execute only once during deployment is crucial. This is particularly important to prevent unnecessary re-execution when new instances are launched by auto-scaling mechanisms.

This document outlines an approach using AWS Systems Manager (SSM) Parameter Store to track the execution state of critical deployment tasks. The solution ensures that these tasks run only once per deployment, regardless of the number of instances, and do not execute when instances are launched by auto-scaling

How the Solution Works

1. AWS SSM Parameter Store stores a state value (0 or 1) for each task.
2. Deployment scripts check the state before executing the task:
   • If the value is 0, the task runs and updates the state to 1 to prevent re-execution.
   • If the value is 1, the script skips execution.
3. AWS CodePipeline resets the state to 0 at the start of each new deployment to allow execution once.
    

Benefits.                                           

• Ensures One-Time Execution: Prevents redundant execution of critical deployment tasks.
• Seamless Auto-Scaling: Avoids unnecessary task execution when new instances are added via auto-scaling.
• Improved Stability: Reduces risk of conflicts and errors caused by multiple instances trying to perform the same task simultaneously.
• Automated Reset: Ensures that deployment tasks run at the start of each deployment via a pipeline step.
   

Implementation Details.                  

1. Use Case: Odoo Module Upgrade

Modify the existing module upgrade script to incorporate state tracking using AWS SSM Parameter Store.

Script: module_upgrade.sh

#!/bin/bash

# Store the parameter name

PARAMETER_NAME="/parameter/module-upgrade-state"

# Get the current parameter value

PARAM_VALUE=$(aws ssm get-parameter --name "$PARAMETER_NAME" --query "Parameter.Value" --output text)

# Check if command was successful

if [ $? -ne 0 ]; then

    echo "Error: Failed to retrieve parameter value"

    exit 1

fi


# Check if value is 0 (indicating upgrade needs to run)

if [ "$PARAM_VALUE" = "0" ]; then

    aws ssm put-parameter \

     --name "$PARAMETER_NAME" \

     --value "1" \

     --type String \

     --overwrite


    echo "Executing Odoo Module Upgrade..."

    log_file=/var/log/odoo/odoo.log

    echo "Starting Module Upgrade..." | tee -a $log_file

    chown odoo:root /var/log/odoo/odoo.log
    
    python3 /opt/odoo/odoo-bin -c /etc/odoo.conf -u all --stop-after-init &

    pid=$!

    echo "Waiting for module upgrade to complete..." | tee -a $log_file

    wait $pid

    echo "Module Upgrade Complete." | tee -a $log_file
    

    if [ $? -ne 0 ]; then

        echo "Error: Failed to update parameter value"

        exit 1

    fi

    echo "Parameter value updated to 1"

else

    echo "Skipping Module Upgrade (parameter value is $PARAM_VALUE)"

fi

2. Use Case: Database Migration

For applications requiring database migrations, the same approach can be used to ensure that migrations run only once per deployment.

Script: db_migration.sh

#!/bin/bash

# Store the parameter name

PARAMETER_NAME="/parameter/db-migration-state"

# Get the current parameter value

PARAM_VALUE=$(aws ssm get-parameter --name "$PARAMETER_NAME" --query "Parameter.Value" --output text)

# Check if command was successful

if [ $? -ne 0 ]; then

    echo "Error: Failed to retrieve parameter value"

    exit 1

fi

# Check if value is 0 (indicating migration needs to run)

if [ "$PARAM_VALUE" = "0" ]; then

    aws ssm put-parameter \

     --name "$PARAMETER_NAME" \

     --value "1" \

     --type String \

     --overwrite

    echo "Executing Database Migration..."

    log_file=/var/log/db_migration.log

    echo "Starting Database Migration..." | tee -a $log_file

    chown root:root /var/log/db_migration.log   

    # Database migration command here

    python3 manage.py migrate &

    pid=$!

    echo "Waiting for database migration to complete..." | tee -a $log_file

    wait $pid

    echo "Database Migration Complete." | tee -a $log_file

    if [ $? -ne 0 ]; then

        echo "Error: Failed to update parameter value"

        exit 1

    fi

    echo "Parameter value updated to 1"

else

    echo "Skipping Database Migration (parameter value is $PARAM_VALUE)"

fi

4. Reset Parameter Before Deployment

To ensure that deployment tasks run only once per deployment, the parameters must be reset at the start of each deployment. This is achieved by adding a build step in the CI/CD pipeline.

Build Step Command in CodePipeline examples

aws ssm put-parameter \

--name "/parameter/module-upgrade-state" \

--value "0" \

--type String \

--overwrite

aws ssm put-parameter \

--name "/parameter/db-migration-state" \

--value "0" \

--type String \

--overwrite

Breakdown of Scripts

Each script follows the same structure:

1. Checking the Parameter Value

PARAMETER_NAME="/parameter/module-upgrade-state"

PARAM_VALUE=$(aws ssm get-parameter --name "$PARAMETER_NAME" --query "Parameter.Value" --output text)

Retrieves the current state value (0 or 1).

If retrieval fails, it exits to prevent unexpected behavior.

2. Running the Task Only if Needed

if [ "$PARAM_VALUE" = "0" ]; then

If the value is 0, the task executes.

3. Updating the Parameter Store

Marks the task as completed to prevent re-execution.

aws ssm put-parameter --name "$PARAMETER_NAME" --value "1" --type String --overwrite

The parameter is immediately updated to 1 to ensure other instances skip execution.

4. Performing the Task

Each script has a specific command for the required task:

Odoo Module Upgrade: Runs odoo-bin upgrade command.

Database Migration: Runs manage.py migrate.

5. Skipping Execution If Already Run

else
    echo "Skipping Task (parameter value is $PARAM_VALUE)"
fi

Ensures that redundant execution does not happen.

Resetting the State for the Next Deployment

To allow the tasks to run again in the next deployment, AWS CodePipeline includes a build step:

aws ssm put-parameter --name "/parameter/module-upgrade-state" --value "0" --type String --overwrite

This ensures that the scripts will execute once when the next deployment occurs.

5. AppSpec.yml for AWS CodeDeploy

To integrate these scripts into AWS CodeDeploy, an appspec.yml file can be used to define lifecycle hooks.

AppSpec File: appspec.yml

version: 0.0
os: linux
files:
  - source: ./
    destination: /opt/application
hooks:
  ApplicationStart:
    - location: /opt/application/module_upgrade.sh
      timeout: 300
      runas: root

This appspec.yml file ensures that the module upgrade scripts run during the ApplicationStart lifecycle event of AWS CodeDeploy.

Conclusion

By leveraging AWS SSM Parameter Store and AWS CodeDeploy, we ensure that critical deployment tasks such as Odoo module upgrades, database migrations, and data seeding run only once per deployment. This approach improves efficiency and reliability in multi-instance deployments while supporting auto-scaling.