9 min readConfiguration management in Python like a Boss(Pydantic with python-dotenv)

Python, Tech / May 8, 2022 / By

The most basic thing of any application is the configuration variables. Some of them are constant throughout the application lifecycle and others keep on changing. The ones that are constants we just create a constant package and reference them from it. It helps us to avoid all the hardcoding in the application. And the other part that is most important is the variables that are kept on changing in the application that is generally known as Configurations.

So how to manage them, where to store them, how to access them? There are many questions related to configuration management. In this blog, we will be going to discuss the ways to manage configuration in Python. Especially the pydantic library

Will Smith Clap GIFfrom Will Smith GIFs

Overview

There are several ways of managing configurations. Let’s talk about the most common ones

Constants

Store the configurations as a constant in the application and they will be applied once you start your application. This is the easiest way but the major disadvantage is that whenever you want to change the configuration you have to redeploy the app again. Also to change the configurations you have to change the code every now and then.

File-based Configurations

There are many ways to do it like using a .ini, JSON, XML, YAML file for storing configuration and using libraries like config parser or loading JSON config in frameworks like Django and Flask. But if it comes to adding more configuration or overriding them or putting some logic into it, then it’s difficult for them

Environment Variables

This is the most common way nowadays as you just have to load the configuration using environment variables or environment files. It works mostly the same way as file-based configuration but the overriding capability is insane.

In this blog, we are going to talk mostly about configuration management using environment variables.

Let’s dive into it

Danske Gifs Mickøgendahl GIFfrom Danske Gifs GIFs

Config Management with Environment Variables

Let’s create a simple flask Application

First, create a config file and named it config.py

config.py

Insert a configuration NAME in it and load it with the environment variable

# config.py
import os

NAME = os.environ.get('NAME')

main.py

Access the same environment variable inside the APP

# main.py
from flask import Flask
import config
app = Flask(__name__)


@app.get("/")
async def hello_world():
    template = f'''
    <p>Hello, {config.NAME}!</p>
    '''
    return template

if __name__ == '__main__':
    app.run(port=5050)

Now to expose the environment variable to the app there are several ways

1. Inline

NAME=ANAND python main.py

2. Export the Environment Variable

export NAME=ANAND
python main.py

3. Load from a .env file

Create a .env file

.env

# .env
NAME=ANAND

If you want to load the whole .env file from the local system then create another file

run_with_env.sh

ENV_FILE="$1"
CMD=${@:2}

set -o allexport
source $ENV_FILE
set +o allexport

$CMD

change this file to an executable file

chmod 755 run_with_env.sh

Now load the .env file using the below command

./run_with_env.sh .env python main.py

4. python-dotenv

Install the library using the below command

pip install python-dotenv

So before initializing the app you just have to put these lines

from dotenv import load_dotenv

load_dotenv()  # take environment variables from .env.

And you’re main.py will look like this now

from flask import Flask
import config
from dotenv import load_dotenv

load_dotenv()  # take environment variables from .env.

app = Flask(__name__)


@app.get("/")
async def hello_world():
    template = f'''
    <p>Hello, {config.NAME}!</p>
    '''
    return template

if __name__ == '__main__':
    app.run(port=5050)

Run the application without exporting the variables explicitly by the export command

python main.py

This is the basic usage of environment variables. Hope everyone is with me till now and you must be using these ways already as there is nothing new in it.

I Completely Agree I Agree GIFfrom I Completely Agree GIFs

Now let’s talk about managing the environment using the Pydantic library

Pydantic

According to Pydantic’s official documentation

Data validation and settings management using python type annotations.Pydantic enforces type hints at runtime and provides user-friendly errors when data is invalid. Define how data should be in pure, canonical python; validate it with pydantic.

One of the most useful applications of Pydantic is setting management. So let’s use Pydantic in our application.

For setting management, we have to create a custom config class and have to inherit BaseSettings class of Pydantic. Let’s modify our configuration to have a different environment and some more configurations.

1. Class-based config

config.py

import os
from typing import Union

from pydantic import BaseSettings


class Base(BaseSettings):
    SECRET_KEY: str = "random_string"
    PORT: int = 5050
    USERNAME: str = "ANAND"settings = Base()

Now with the above configuration, you can access your configuration using settings.PORT like that. As you can see it also support type annotation that will really help you for type checking.

So if you do

print(settings.SECRET_KEY)
# Output "random_string"

It will just display you the default value that we have mentioned in the configuration.

2. Environment variable combined with Class-based configuration

let’s say if you want to provide some other random value on the go to the SECRET_KEY. Then you can pass the environment variable for that. So let’s say we do

export SECRET_KEY='env_random_string'

Now do

print(settings.SECRET_KEY)
# Output "env_random_string"

So environment variable automatically overrides the secret key. So most of the time it’s the case that your environment variable is having a different name than the config name, which is also possible with Pydantic. Just change the configuration like below

SECRET_KEY: str = Field('random_string', env='ANOTHER_SECRET_KEY')

Here we have used Field the variable of Pydantic. We can provide env parameters to specify from which environment variable we have to take the configuration from

export ANOTHER_SECRET_KEY=another_env_key# Run the application print(settings.SECRET_KEY)
# Output "another_env_key"

3. Env based Configuration using Inheritance

We can also use Pydantic for our production-based application where there are different environments.

config.py

import os
from typing import Union

from pydantic import BaseSettings, Field


class Base(BaseSettings):
    SECRET_KEY: str = Field('random_string', env='ANOTHER_SECRET_KEY')
    PORT: int = 5050
    USERNAME: str = "ANAND"


class Dev(Base):
    USERNAME = "TRIPATHI"


class Prod(Base):
    USERNAME = "Production"
    PORT = 5051


config = dict(
    dev=Dev,
    prod=Prod
)
settings: Union[Dev, Prod] = config[os.environ.get('ENV', 'dev').lower()]()

Just run the application using

In the main application, you can use settings variables to access all the configurations

from flask import Flask

from dotenv import load_dotenv

from config import settings

load_dotenv()  # take environment variables from .env.

app = Flask(__name__)


@app.get("/")
async def hello_world():
    template = f'''
    <p>Hello, {settings.USERNAME}!</p>
    '''
    return template

if __name__ == '__main__':
    app.run(port=5050)

4. Case sensitive environment variables

Most of the time our environment variables are CAPITAL cases and our configurations are smaller cases, so in that case, we cannot override every configuration with another name. So that can be fixed using the case-sensitive property. Just add a metaclass Config inside settings classes

class Base(BaseSettings):
    secret_key: str = Field('random_string', env='ANOTHER_SECRET_KEY')
    port: int = 5050
    username: str = "ANAND"

    class Config:
        case_sensitive = False

Now the username configuration is getting the value from USERNAME/username environment variable.

5. Dotenv (.env) support

The crazy feature of Pydantic is it is having out of box full support of the python-dotenv library. You can actually load the whole .env file into the python application context. You just have to install Pydantic with an extra dependency of python-dotenv like below

pip install pydantic[dotenv]

Now let’s update our old .env file with the updated value of the configuration

ANOTHER_SECRET_KEY=anything
PORT=5080
USERNAME=Test

Jut modify the configuration so that it will get configuration from .env file

class Base(BaseSettings):
    secret_key: str = Field('random_string', env='ANOTHER_SECRET_KEY')
    port: int = 5050
    username: str = "ANAND"

    class Config:
        case_sensitive = False
        env_file = '.env' # This is the key factor

Now your base settings will read all the environment variables from .env. In a real-world scenario, we are having different environment variables for dev different for prod and different for other environments. We can achieve this by pydantic like below.

class Base(BaseSettings):
    secret_key: str = Field('random_string', env='ANOTHER_SECRET_KEY')
    port: int = 5050
    username: str = "ANAND"

    class Config:
        case_sensitive = False
        env_file = '.env'


class Dev(Base):
    username = "TRIPATHI"
    
    class Config:
        env_file = 'dev.env'


class Prod(Base):
    username = "Production"
    port = 5051
    
    class Config:
        env_file = 'prod.env'

However, there are many features of Pydantic that I cannot cover here but I guess this was sufficient to convince you all to use this library for configuration management. If still you are not convinced you can go to this official doc of Pydantic Configuration Management where you will find some more features of it.

To access the full project please check out this Github link

.env

ANOTHER_SECRET_KEY=base
PORT=5080
USERNAME=base

dev.env

ANOTHER_SECRET_KEY=dev
PORT=5080
USERNAME=DevUsername

prod.env

ANOTHER_SECRET_KEY=prod
PORT=5090
USERNAME=ProdUsername

config.py

main.py

Now if you run ENV=dev python main.py or python main.py It will output

It will run on port 5080 and the output is

If you run with a production environment like ENV=prod python main.py then it will run on port 5090 and the output is

Conclusion

I hope you will use this method of configuration management in your local and production applications and a plus point is that it will move your app more towards a 12-factor application.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.