Think with Enlab

Diving deep into the ocean of technology

Stay Connected. No spam!

How to Secure Sensitive Data in Configuration

 

In the blog post “How to Configure .Net Core Environments With Practical Examples”, we shared the .NET Core Environment Configuration with the appsettings.json file.

As you can see, the appsettings.json file helps us to store the configuration, the setting, etc. However, there is a big concern that all information on this file can be accessed by anyone who has the source code or the server that hosted the app. For this reason, it’s not a good place to store sensitive data.

But what is sensitive data? Sensitive data is anything that can be potentially exploited by a third party. For example, API keys, connection strings, tokens, username, emails, passwords, password hashes, potentially some URLs, etc. In this article, I would like to share with you how to secure sensitive data in local and production environments.

 

How to secure sensitive data in the development environment (Locally)

In the development environment, sensitive data should not be stored in configuration files, much less in the source code itself. But it should be easy to be updated.

You can choose either to store your configuration in the environment variables or as a user secret by using the Secret Manager. Neither of these options is good enough for production since they store your data as plain texts instead of encrypting the data. However, you can use them to avoid conflicts while working with other developers on a project. By doing this, you don’t have to worry if secrets and environment variables are committed by mistakes.

 

Environment variables.

According to Wikipedia, an environment variable is a dynamic-named value that can affect the way running processes will behave on a computer.

For Windows in CMD: you can use the “set” command to set an environment variable with the syntax like this:


set [variable name]=[value]Code language: C# (cs)

 You can check the full options for set commands.

Example:  


    set ConnectionStrings_ProjectConnectionString=server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;
Code language: C# (cs)

 

And then you can use “set” without any arguments to list all environment variables.

 

set command examples

 

For  PowerShell: 

 


    [System.Environment]::SetEnvironmentVariable('[variable name]','[value]',[System.EnvironmentVariableTarget]::User)
Code language: C# (cs)

 

Example: The same command with the above CMD is: 

 


    [System.Environment]::SetEnvironmentVariable('ConnectionStrings_ProjectConnectionString','server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;',[System.EnvironmentVariableTarget]::User)
Code language: C# (cs)

 

You can check your variable with:

 


    [System.Environment]::GetEnvironmentVariable('[variable name]','user')
Code language: C# (cs)

system.environment example

 

If you need to read the environment variable programmatically, please refer to “How to Configure .Net Core Environments With Practical Examples”. It also includes the tutorial to manage environment variables with the windows GUI.

 

Secret Manager

.NET Core provides a secrets manager tool to store sensitive data during the development. This tool allows you to store your sensitive data locally on your machine, outside the project tree.

To start using secrets, you need to enable them. Go to the project directory and enter the command:

 


    dotnet user-secrets init
Code language: C# (cs)

dotnet user-secrets inits 1

 

As the result, the command creates an entry in the .csproj file:

 

dotnet user-secrets inits 2

 

Now to create or modify a secret, you can use the dotnet user-secrets set command as follows:


    dotnet user-secrets set "ConnectionStrings_ProjectConnectionString" "server=\\DEV-SERVER; database=DatabaseName; User Id=myaccount; Password=mypassword;"

Code language: C# (cs)

 

And then use the below “list” command to check if you’ve successfully added it:


    dotnet user-secrets list
Code language: C# (cs)

 

Here is the result:

 

dotnet user-secrets list

 

To access the user secret in the .NET Core project, you will need to add some additional codes for ConfigurationBuilder as below:

 

add additional codes for ConfigurationBuilder to access user secrets in .net core project

 

To remove a secret you can use “dotnet user-secrets remove [secret key]”, and use “dotnet user-secrets clear” to remove all.

But if you are using Visual Studio you can easily manage the user secret by GUI. Just right-click on the project and choose “Manage User Secrets”.

 

manage user secrets in Visual Studio

 

The secrets.json file will be opened with all your secrets settings. Now, you can edit and save it just like the “appsettings.json” file.

This secrets.json file is not super secure, and the keys are not encrypted. However, it provides an easy way to avoid storing secrets in your project config files and having to remember to add them to the source control ignore list.

 

Using the Azure Key Vault to secure sensitive data in the production environment

Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys. You can change the secret directly from the Azure portal and no need to re-deploy your application.

Setting up Azure key vault

To setup key vault on the Azure portal, you need to follow the following steps:

1. From the Azure portal menu or the Home page, select Create a resource

2. In the search box, enter Key Vault

3. From the results list, choose Key Vault

 

Create key vault in Azure portal 1

 

4. On the Key Vault section, choose to Create

5. On the Create key vault section, provide the following information

 

 

  • Subscription: choose a subscription.
  • Resource group: choose to create new and enter a resource group name.
  • Key vault name: a unique name is required.
  • Region: choose your region.
  • Pricing tier: check the pricing for the Azure key vault here.

Note: Leave the other options to their defaults.

6. Click on “Review + create” to finish your setting.

 

Create key vault in Azure portal 3

 

Now, let’s add your secrets. You can refer to this Azure Quickstart for the details of how to create an Azure key vault and add a secret with the Azure portal.

 

Connecting .NET Core app with Azure key vault

To connect your .NET Core app with the Azure key vault, you will need to do some further steps:

 

add configuration on the program.cs file

 

The “https://contoso-vault2.vault.azure.net/” is the “DNS Name” on your Azure key vault overview page.

 

example of DNS Name on Azure key vault overview page

 

If your application is hosted on Azure. You can easily manage access to the Azure key vault on the Azure portal under “Access policies”.

If you don’t use Azure you can follow this Microsoft document to config the IP firewall or certificate to connect the Azure key vault from outside of the Azure network.

Once the app is connected with the Azure key vault, you can update the Azure Key Vault without updating and redeploying the application. 

However, you need to restart the app to refresh the Key Vault cache since the application caches the secret from the key vault on the application startup. Though, it is not a good idea. 

Fortunately, we can specify a ReloadInterval to reload the secrets from the Key Vault whenever the Reload Interval duration is over. Just update the config for the above Azure key vault like this:

 

add additional codes for ConfigurationBuilder to access user secrets in .net core project

 

Using AWS Systems Manager Parameter Store

Like Azure Key Vault, AWS Systems Manager Parameter Store (let call AWS Parameter Store) provides secure, hierarchical storage for configuration data management and secrets management. Data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes can be stored as parameter values. And values can be stored as plain text or encrypted data.

 

Setting up AWS Parameter Store

To set up AWS Parameter Store, you need to take the following steps:

1. Go to AWS Management Console > find the “System Manager” service

 

AWS management console

 

2. On the “Systems Manager” screen, choose the “Parameter Store” under the “Shared Resources.”

 

AWS system manager

 

3. Click on the “Create Parameter” button

 

AWS system manager parameter store

 

4. On the Create Parameter Store screen, enter the name, parameter type, and  KMS info > click on the “Create Parameter” button.

 

AWS system manager - parameter details

 

5. Now, add your secrets to AWS Parameter Store. If you use AWS console, refer to this Create a Systems Manager parameter (console). If you use Windows PowerShell, use the command as below:


    Write-SSMParameter `

    -Name "parameter-name" `

    -Value "parameter-value" `

    -Type "SecureString"  `

    -KeyId "a KMS CMK ID, a KMS CMK ARN, an alias name, or an alias ARN" `

    --tags "Key=tag-key,Value=tag-value"

Code language: C# (cs)

 

Connecting the .NET Core app with AWS Parameter Store

First, make sure that you install Amazon.Extensions.Configuration.SystemsManager NuGet package which is a configuration provider for the .NET Core configuration system to connect your .net core app with AWS Parameter Store.

Next, like the Azure Key Vault, you need to update your app configuration to use the AWS Parameter Store provider. Take a look at the below picture to see how to update your Program.cs file:

 


    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, config) =>
            {
                if (context.HostingEnvironment.IsProduction())
                {
                    var azureServiceTokenProvider = new AzureServiceTokenProvider();

                    var keyVaultClient = new KeyVaultClient(
                        new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));


                    var keyVaultOptions = new AzureKeyVaultConfigurationOptions
                    {
                        Client = keyVaultClient,
                        Manager = new DefaultKeyVaultSecretManager(),
                        ReloadInterval = TimeSpan.FromMinutes(5) // Reload in a minute
                    };

                    config.AddAzureKeyVault(keyVaultOptions);
                }

            })
            .ConfigureAppConfiguration(builder =>
            {
                builder.AddSystemsManager("/myapplication");
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup();
            });


Code language: C# (cs)

Here, we add all of the parameters from Parameter Store starting with “/myapplication” we create on the setup AWS Parameter Store step.

Like the Azure Key Vault, we also can set the reload interval time, certificate, caching, etc. You can check the details in the .NET Core configuration provider for AWS Systems Manager.

 

Creating custom configuration providers

Besides the existing configuration providers, and 3rd providers like Azure Key Vault, AWS  Parameter Store, etc, we can build custom configuration providers. 

Many software architectures recommend storing configuration sources reside in separate stores to decouple the configurations from the application. Then, send an HTTP request or query the database to load those configurations. About this matter, custom configuration providers will help us to do it easily. But how to create a custom provider? Check this out!

To create a custom provider, we need to implement the IConfigurationSource and IConfigurationProvider interfaces. 

Create configuration source

Preparing the required information to build a configuration provider is the main responsibility of a configuration source. A configuration source needs to implement the IConfigurationSource interface. This interface requires us to implement the IConfigurationProvider Build (IConfigurationBuilder builder) method.

Below is an example of an implementation of a configuration source. This MyCustomConfigurationSource class has two properties including ConnectionString and Query. They are required to determine the SQL Server database connection and the key-value pairs retrieval query.

 


using Microsoft.Extensions.Configuration;
 
namespace WebApplication.Configurations{
 
    public class MyCustomConfigurationOptions    {
 
        public string ConnectionString { get; set; }
 
        public string Query { get; set; }
    }
    public class MyCustomConfigurationSource : IConfigurationSource    {
 
        public string ConnectionString { get; private set; }
 
        public string Query { get; private set; }
 
        public MyCustomConfigurationSource(MyCustomConfigurationOptions options)
        {
            ConnectionString = options.ConnectionString;
 
            Query = options.Query;
        }
        public IConfigurationProvider Build(IConfigurationBuilder builder)        {
 
            return new MyCustomConfigurationProvider(this);
        }
    }
}
Code language: C# (cs)

 

Create configuration provider

Loading the configuration data from a configuration source is the main responsibility of a configuration provider. A configuration provider must implement the IConfigurationProvider interface. However, the fact is the configuration provider class can inherit from the ConfigurationProvider base class which has implemented all methods in the IConfigurationProvider interface. Simply by doing this, we can make use of available codes and skip some overlapped steps in implementing many commonly used methods such as a method to set a value for a key, a method to find a value with a given key, etc.

The simple custom provider to load the setting on the SQL Server database will be like this:

 


using Microsoft.Extensions.Configuration;
using System.Data.SqlClient;
namespace WebApplication.Configurations{
 
    public class MyCustomConfigurationProvider : ConfigurationProvider{
 
        private MyCustomConfigurationSource Source { get; }
 
        public MyCustomConfigurationProvider(MyCustomConfigurationSource source){
 
            Source = source;
 
        }
 
        public override void Load(){
 
            using (var connection = new SqlConnection(Source.ConnectionString)){
 
                connection.Open();
 
                using (var command = new SqlCommand(Source.Query, connection)){
 
                    using (var reader = command.ExecuteReader()){
 
                        while (reader.Read()){
 
                            Set(reader.GetString(0), reader.GetString(1));
 
                        }
 
                    }
 
                }
 
                connection.Close();
            }
 
        }
 
    }
Code language: C# (cs)

 

The constructor of the MyCustomConfigurationProvider class takes an instance of MyCustomConfigurationSource as the parameter. We override the Load() method with our custom implementation from lines 17 to 31. In this method, we query the database using the native .NET SQL Server client from the System.Data.SqlClient NuGet package. We can further add some codes to retry the database connection, handle exceptions, and decrypt the encrypted configuration data if needed.

 

Add the custom configuration provider to the .NET core configuration system

Once the configuration source and the configuration provider are ready to use, we can add them to the .NET core configuration system. Commonly, we use an extension method to add the configuration source to the configuration builder.

 


using Microsoft.Extensions.Configuration;
using System;
 
namespace WebApplication.Configurations
{
    public static class ConfigurationExtensions
    {
        public static IConfigurationBuilder AddMyCustomConfiguration(this IConfigurationBuilder configuration, Action options)
        {
            if (options == null)
                throw new ArgumentNullException(nameof(options));
 
            var configurationOptions = new MyCustomConfigurationOptions();
 
            options(configurationOptions);
            configuration.Add(new MyCustomConfigurationSource(configurationOptions));
 
            return configuration;
        }
    }
}

Code language: C# (cs)

 

And then, use it.

 


public Startup(IWebHostEnvironment environment)
    {
        var builder = new ConfigurationBuilder();
        builder.SetBasePath(environment.ContentRootPath)
        .AddJsonFile("appsettings.json", false, true)
        .AddMyCustomConfiguration(options =>
        {
            options.ConnectionString = Environment.GetEnvironmentVariable("DEV_DB_CONNECTION_STRING");
            options.Query = "SELECT [Key],[Value] FROM [dbo].[Settings]";
        })
        .AddEnvironmentVariables();

        if (environment.IsDevelopment())
        {
            builder.AddUserSecrets();
        }

        Configuration = builder.Build();
    }
Code language: C# (cs)

 

I stored the SQL Server connection string in the environment variable “DEV_DB_CONNECTION_STRING”. Here is how the database for the Settings table looks like:

 

Setting table database

 

One thing to notice here, the setting on the custom provider will override the appsettings.json file if it duplicates because we call the ‘AddMyCustomConfiguration’ method after ‘AddJsonFile("appsettings.json")’.  This rule is the same for the ‘AddEnvironmentVariables’. It’s how the .NET core loads the configuration in case we have multiple configuration providers. The last provider will override all loaded providers. Thus, we must pay attention to the priorities of the providers.

 

Final Thoughts,

As you can see, there are several options for securing sensitive configuration data, each with different levels of security and ease of use. No matter which option you ultimately select, follow these basic guidelines:

  • never commit secrets or sensitive data to a code repository, even a private repository
  • do not store secrets or sensitive data in source code
  • use an encryption mechanism for your data if you can.

 

CTA Enlab Software

 

 

References

About the author

Tan Nguyen

Hi, my name is Tan! I am currently working as a Software Engineer at Enlab Software and I mostly focus on Software Architectures, Development, and Software Re-factoring. I love coding and learning new technologies so I'm excited to share my knowledge with you!

Up Next

How to apply SOLID principles with practical examples in C#
September 07,2021 by Chuong Tran
In Object-Oriented Programming (OOP), SOLID is one of the most popular sets of design principles...
Domain-Driven Design in ASP.NET Core applications
May 31,2021 by Loc Nguyen
The technology industry has been thriving for the second half of the last century. It's...
How to create real time chat applications using WebSocket APIs in API Gateway
March 26,2021 by Trong Pham
My experience developing real-time messaging applications leveraging AWS services inspires me to share with the...
How to build and deploy a 3-tier architecture application with C#
March 11,2021 by Uyen Luu
Hi, back to the architecture patterns, in the last article I explained what the three-layer...
Roll to Top

Can we send you our next blog posts? Only the best stuffs.

Subscribe