Project 12 : GitLab CI/CD for Seamless Website Delivery with Terraform Magic - S3, CloudFront, and ACM Unleashed

Introduction:

🚀 Dive into the wonders of TerraSync Forge – a place where Terraform mastery collides with AWS magic! Struggling with intricate hosting setups? Say goodbye to complexity as we unveil the secrets of effortless static website deployment on S3, swift content delivery through CloudFront, and security reinforcement using ACM. 🌐✨ Immerse yourself in the charm of seamless web hosting and let us navigate you through a world where hosting challenges disappear effortlessly! 🏰

Why This Project Matters?

In my last project, using Terraform to set things up got a bit complicated. The files we needed to work together were stuck on my computer, making it tough for others to join in. Plus, whenever we made changes to the Terraform stuff or wanted to freshen up the website, we had to handle some commands manually. It was a bit of a hassle.

That's where this project comes in—to make things easier. With everything set up through GitLab CI/CD, collaboration is a breeze. No more files locked away on one computer—now, we can all team up seamlessly. Changes to Terraform or website content? No problem, it all gets sorted out automatically. It's teamwork without the fuss of complicated commands, all thanks to this project! 🚀✨

Technologies Used in This Project:

1. Terraform: Utilized for defining and provisioning infrastructure as code.

2. GitLab CI/CD: Employs continuous integration and continuous deployment for streamlined development workflows.

3. AWS (Amazon Web Services):

   • S3 (Simple Storage Service): Storage for website content.

   • CloudFront: Content delivery network for accelerated website performance.

   • ACM (AWS Certificate Manager): Manages SSL/TLS certificates for secure connections.

Project Overview:

Prerequisites:

1. Terraform Installed: Make sure Terraform is installed on your machine. Download it here and follow the installation guide.

2. AWS Account: Have an active AWS account ready with the necessary permissions to create S3 buckets, CloudFront distributions, and ACM certificates.

3. Your Ready-to-Launch Static Website: Prepare your static website content, ready to take off into the cloud.

4. A Domain configured with Route53 records.

Project Initialization Steps:

To kickstart the project, follow these straightforward setup instructions:

1. Configure GitLab Variables:

   • Navigate to your project settings, then select "CI/CD," and click on "Variables."

• Add your AWS access key ID and secret access key with minimal permissions.

  

1. Generate GitLab Access Token:

   • Head to your GitLab profile, click on "Edit Profile," then navigate to "Access Tokens" and generate a personal access token.

     

2. Add Variables:

   • In GitLab Variables, include the following:

     • AWS Access Key ID

     • AWS Secret Access Key

     • GitLab Access Token (Generated in the previous step)

     • Username (Simply use your GitLab username)

We need to add .gitlab-ci.yml in root directory of gitlab project.

---
workflow:
    rules:
      - if: $CI_COMMIT_BRANCH != "main" && $CI_PIPELINE_SOURCE != "merge_request_event"
        when: never
      - when: always

variables:
  TF_DIR: ${CI_PROJECT_DIR}/terraform
  STATE_NAME: "ajaycodelab-tf"
  ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${STATE_NAME}"

stages:
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:light
  entrypoint: [""]

before_script:
  - terraform --version
  - export GITLAB_ACCESS_TOKEN=$TOKEN
  - export GITLAB_USERNAME=$USERNAME
  - cd ${TF_DIR}
  - apk --no-cache add bash py-pip groff less mailcap jq
  - pip install --upgrade pip
  - pip install awscli
  - terraform init  -reconfigure -backend-config="address=${ADDRESS}" -backend-config="lock_address=${ADDRESS}/lock" -backend-config="unlock_address=${ADDRESS}/lock" -backend-config="username=$GITLAB_USERNAME" -backend-config="password=$GITLAB_ACCESS_TOKEN" -backend-config="lock_method=POST" -backend-config="unlock_method=DELETE" -backend-config="retry_wait_min=5"

validate:
  stage: validate
  script:
    - terraform validate
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull-push

plan:
  stage: plan
  script:
    - terraform plan 
  dependencies:
    - validate
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull


apply:
  stage: apply
  script:
    - terraform output
    - export S3_BUCKET_NAME=$(terraform output -json | jq -r '.s3_bucket_name.value')
    - "echo \"S3 Bucket Name: $S3_BUCKET_NAME\""
    - |
      if [ -n "$(git diff --name-only $CI_COMMIT_BEFORE_SHA...$CI_COMMIT_SHA | grep 'terraform/')" ]; then
        terraform apply -auto-approve 
      elif [ -n "$(git diff --name-only $CI_COMMIT_BEFORE_SHA...$CI_COMMIT_SHA | grep 'website/')" ]; then
        aws s3 sync ${CI_PROJECT_DIR}/website s3://$S3_BUCKET_NAME/
      else
        echo "No changes in 'terraform/' or 'website/' directory. Nothing to apply or sync."
      fi

  dependencies:
    - plan
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull

destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    - plan
    - apply
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull
  when: manual

Our project directory will look like this :

Organize all Terraform files in the "Terraform" folder, used in our previous project.

After initializing the project repo, run this command in your Terraform directory to generate the state file on GitLab:

terraform init -reconfigure \
  -backend-config="address=https://gitlab.com/api/v4/projects/<your project id>/terrafrom/state/" \
  -backend-config="lock_address=https://gitlab.com/api/v4/projects/<your project id>/terraform/state/ajaycodelab-tf/lock" \
  -backend-config="unlock_address=https://gitlab.com/api/v4/projects/<your project id>/terraform/state/ajaycodelab-tf/lock" \
  -backend-config="username=<your gitlab username>" \
  -backend-config="password=<yout gitlab token>" \
  -backend-config="lock_method=POST" \
  -backend-config="unlock_method=DELETE" \
  -backend-config="retry_wait_min=5"

Upon successful configuration, you'll see an output confirming the backend setup.

To verify, navigate to "Operate" and "Terraform States" on GitLab to find your generated state file. GitLab doesn't generate the state file automatically, so this step is crucial.

That it. we have configured our project. It's time to push out code to Gitlab and see the magic.

It will start pipeline automatic.

After completion of pipeline , you will see success message and deployed website on your S3 bucket.

Problem faced and How I solved:

Building the pipeline turned out to be quite a head-scratcher. I found myself searching a lot for each step. The trickiest part came when my git push would kick off the pipeline, but the website content didn't update because my Terraform state file remained unchanged. After some deep digging, I figured out a solution and I developed this logic:

If anything changes in the Terraform folder, it triggers a terraform apply. On the flip side, if there's a tweak in the website folder, it syncs up with the S3 bucket. This simple logic ensures that the pipeline does exactly what it's supposed to do.

Gitlab repo If you want to copy it: https://gitlab.com/learning1482041/cloudforge.git

Conclustion:

Before this project, setting up infrastructure and updating website content was a bit of a puzzle. The Terraform state file stayed stuck on individual computers, making teamwork challenging.

Enter our solution. This project simplifies everything. It automates infrastructure setup with Terraform, freeing up the state file for teamwork. GitLab CI/CD takes charge of the pipeline, making sure changes in the Terraform folder trigger an update, and tweaks in the website folder effortlessly sync with the S3 bucket.

In a nutshell, this project makes hosting setups a breeze. With a mix of Terraform, GitLab CI/CD, and AWS, it crafts a deployment process that's not just straightforward but also collaborative and automated—a significant upgrade from the previous cumbersome approach. Hosting challenges are now a thing of the past.

Pro tip :

To save money, I've added a special job to delete everything in the infrastructure. Just give it a manual kick, and you're good to go! 🚀💰