Setting Up Your Cloud Development Environment

In this section we will learn how to set up our cloud development environment in the AWS,GCP and Azure cloud.

All the providers are pretty much following the same procedure :

Go to the Cloud provider Platform and click on the "Create an account" button.
Enter your personal information and create an account.
Once your account is created, you will need to activate it by verifying your email address and enjoy free cloud credit for a given period of time.
Once your account is activated, you can start using the provider services.

GCP Account Setup

Follow the official procedure here

Azure Account Setup

Follow the official procedure here

AWS Account Setup

If you have special request, issues take a look at the official aws account guide here

Python Environment

Installing Python3

Visit the official Python website and download the latest version for your OS. Follow the installation instructions.

How to work with virtual environment

A virtual environment is a sandboxed Python environment that allows you to install and manage Python packages independently of your system Python installation.

This is important because it prevents conflicts between packages that are used by different projects like you can see in the meme below.

There are several reasons why it is important to work with virtual environments in Python:

To isolate project dependencies: Virtual environments allow you to isolate the dependencies of each project from each other and from your system Python installation. This means that you can install different versions of the same package for different projects without causing any conflicts.
To avoid conflicts with system packages: Virtual environments also help to avoid conflicts between Python packages that are installed in your system Python installation and Python packages that are installed in a virtual environment.
To create reproducible environments: Virtual environments allow you to create reproducible environments for your Python projects. This is useful for development, testing, and deployment.

Conda

Conda is an open-source package manager and environment management system. It allows users to install multiple versions of software packages and their dependencies, and switch between them. Conda is especially popular in the data science and machine learning communities because it makes it easy to manage complex software stacks.

Manage Conda Environments

Quick recap on how to manage your Conda Environments on Linux and MacOS and avoid situation like this :

Creating a New Environment and specify the version of Python

conda create --name myenv python=3.9

Activating/Deactivating an Environment

source activate myenv

and

source deactivate

Listing all your environments

conda env list

Installing python packages in your environment

First you have to activate the environment, then use the following command to install a package:

conda install numpy

Exporting your environment

If you want to share your environment with others or recreate it on another machine, you can export it to a file call environment.yml:

conda env export > environment.yml

Creating and removing an environment from an existing `.yml` file

conda env create -f environment.yml

and

conda env remove --name myenv

Well, now you know how to work with conda and create your perfect work env 🥳

Move your conda env to an other disk

Sometimes you do not want all your conda env into the same disk, it can be a lot of weight to carry for one disk. Let's say your new SSD is mounted at /mnt/new_ssd and you want to move your Conda environment there. You can use rsync or cp to copy the environment directory:

rsync -av /path/to/old/conda/envs/your_env_name /mnt/new_ssd/path/to/new/conda/envs/

You can retreive the path of your conda env with the list command. Make sure to replace /path/to/old/conda/envs/ with the actual path to your Conda environments on the old disk and /mnt/new_ssd/path/to/new/conda/envs/ with the desired path on the new SSD.

You need to tell Conda about the new environment location. You can do this by editing the .condarc file in your home directory or by using Conda's configuration command. If you're moving all your environments, you can set the envs_dirs option:

conda config --add envs_dirs /mnt/new_ssd/path/to/new/conda/envs/

Then you can activate the environment from the new location to ensure it works correctly:

conda activate /mnt/new_ssd/path/to/new/conda/envs/your_env_name

Once you're sure that everything works correctly from the new location, you can remove the old environment to free up space :

rm -r /path/to/old/conda/envs/your_env_name

Add your conda env into jupyter kernel

First, activate the Conda environment you want to add to Jupyter within your activated environment, install the ipykernel package, which provides the IPython kernel for Jupyter :

pip install ipykernel

Then simplly use the ipykernel module to install your environment as a new kernel :

python -m ipykernel install --user --name your_env_name --display-name "Your Display Name"

You can just lunch a jupyter server in order to verify this is working correctely. That's it! You've now added your Conda environment to Jupyter as a new kernel, and you can use this environment to run Jupyter notebooks 🥳

Bonus conda script

If you often work with conda, you can save time by writing some bash script in order to automate actions like list all your Conda environments, their sizes, and the packages they contain in a formatted manner.

#!/bin/bash

# Determine the format based on the argument
format="line"
if [ "$1" == "list" ]; then
    format="list"
fi

# Get the list of Conda environments
envs=$(conda env list | awk 'NR>3 {print $1}')

# Print header
echo -e "\e[1;34mConda Environments Information\e[0m"
echo "----------------------------------"

# Loop through each environment
for env in $envs; do
    # Get the path of the environment
    env_path=$(conda env list | grep "^$env " | awk '{print $2}')

    # Get the size of the environment
    env_size=$(du -sh $env_path 2>/dev/null | awk '{print $1}')

    # List packages in the environment based on the chosen format
    if [ "$format" == "list" ]; then
        packages=$(conda list -n $env | awk 'NR>3 {print "- " $1}')
    else
        packages=$(conda list -n $env | awk 'NR>3 {print $1}' | tr '\n' ', ' | sed 's/, $//')
    fi

    # Print environment details
    echo -e "\e[1;32mEnvironment:\e[0m $env"
    echo -e "\e[1;32mPath:\e[0m $env_path"
    echo -e "\e[1;32mSize:\e[0m $env_size"
    echo -e "\e[1;32mPackages:\e[0m\n$packages"
    echo "----------------------------------"
done

Conda & Virtualenv what is the differences

While both conda and virtualenv serve the purpose of creating isolated environments, the choice between them often depends on the specific needs of a project.

If you're working on data science projects with complex dependencies, including non-Python packages, conda might be more suitable. If you're working on pure Python projects and want something lightweight, virtualenv might be the way to go.

Cloud overview projects

Let's begin by basic tasks to handle Microsoft Azure and understand its basics:

Ceate a Virtual Machine (VM)

Imagine you need to host a website or run a specific application that requires a Windows/Linux environment. Creating a VM in Azure can provide you with the required environment without needing physical hardware.
- 📚 Understand the fundamentals of cloud computing, learn how to manage VMs, and gain insights into Infrastructure as a Service (IaaS) offerings in Azure.
Set Up Blob Storage

Consider a scenario where your application generates a large number of log files or images that need to be stored and accessed securely. Azure Blob Storage can be used to store such unstructured data.
- 📚 Learn about Azure’s object storage, manage files in the cloud, and understand data accessibility and security.
Deploy a Web App

Suppose you have developed a web application and you need a platform to host it. Azure App Services can provide a managed platform to deploy web apps without managing the underlying infrastructure.
- 📚 Familiarize yourself with Platform as a Service (PaaS), understand web hosting on Azure, and learn about deployment strategies and scaling.
Create a SQL Database

Imagine developing an application that requires a relational database backend. Creating an Azure SQL Database allows you to have a scalable and managed database service.
- 📚 Understand database management in the cloud, learn about scalability and backup options, and explore SQL operations in a managed environment.
Implement a simple Azure Functions

Consider a scenario where you need to process data or respond to events without a dedicated server. Azure Functions allow you to write event-driven functions that can be triggered on-demand.
- 📚 Learn about serverless computing, explore event-driven programming, and understand the benefits of a microservices architecture.
Set Up a Networking ressource

Suppose you have multiple VMs or services that need to communicate securely. Creating a Virtual Network in Azure allows you to isolate and manage network traffic between resources.
- 📚 Understand basic networking concepts in Azure, learn about network isolation and security, and explore communication between cloud resources.
Configure Monitoring and Logging (at least for cost 😂)

Imagine needing insights into the performance and health of your resources. Azure Monitor provides detailed telemetry and logs to help you diagnose issues and optimize performance.
- 📚 Learn about monitoring, logging, and diagnostics in Azure, understand the importance of observability, and explore alerting and notification options.
Implement Azure Identity by creating a simple dev role

Consider a scenario where you need to manage user access to resources. Azure Active Directory allows you to manage identities and control access to resources.
- 📚 Understand identity management, learn about access control and role-based access, and explore user authentication and authorization.
Create a personalized Resource Group

Imagine needing to organize and manage related Azure resources. Resource Groups in Azure provide a way to group resources based on lifecycle, permissions, or other criteria.
- 📚 Learn about resource organization and management in Azure, understand the lifecycle of resources, and explore resource grouping strategies.
Set Up Auto-Scaling

Suppose you have a variable workload and need to adjust resources based on demand. Auto-scaling in Azure allows you to dynamically adjust resources to maintain performance and manage costs.
- 📚 Understand scalability and performance management in Azure, learn about dynamic resource allocation, and explore cost management strategies.