Blog

Flox and AWS: Taking the Chaos Out of Secrets Management

Steve Swoyer | 5 December 2024

So you’re wrapping up this year’s AWS re:Invent, inspired, overwhelmed with new ideas, yet still struggling to process the sheer surfeit of stuff Amazon announced this week. There’s so much you want your team to experiment with, but at the same time you feel blocked by a familiar (and frustrating) challenge.

You still lack a standard, reproducible solution that teams can use to securely manage AWS secrets and authenticate consistently with AWS services, especially in collaborative local development workflows spanning different OSes and machine architectures.

This article walks you through how you can use Flox to create reproducible dev environments for managing AWS secrets, authenticating with AWS services, and automating AWS workflows.

All Flox environments are 100% portable, meaning they’ll run and behave the same way everywhere—in local dev, CI, and production. If you struggle with AWS secrets management in a team-based setting, this walk-through was written with you in mind!

Intrigued? Read on to discover how this works!

Keeping It Like a Secret

For this walk-through, we’ll showcase a Flox environment that combines 1Password, the popular secrets-management service, and the AWSCLIv2, which we’ll assume doesn’t need an introduction.

Secrets management in this environment is handled as follows:

• We authenticate via the 1Password CLI when we first activate our Flox environment;
• We use the 1Password CLI to obtain long-term AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY secrets from the 1Password vault in which they live;
• We use these to obtain short-term secrets from the AWS Secure Token Service (STS);
• We export our short-term secrets as environment variables to an ephemeral 1Password environment inside the Flox environment. This is enabled via 1Password’s op run sub-command.

Got it? Ready to dive in?

Floxifying Secrets Management

If you don’t already have Flox installed, go ahead and download it.

Then pull a copy of my pre-built AWS-1Password example environment from FloxHub and activate it locally:

flox pull --copy barstoolbluz/aws-1pass && flox activate

Or, if you’d rather test drive it first, you can activate it as a remote (i.e., ephemeral) Flox environment:

flox activate -r barstoolbluz/aws-1pass
 
✅ You are now using the environment 'barstoolbluz/aws-1pass (remote)'.
To stop using this environment, type 'exit'
 
No existing 1Password session found. Authenticating...
No accounts configured for use with 1Password CLI.

See those last two lines? The 1Password CLI expects to work with default environment variables and/or read required data from its a pre-populated configuration file. You’ll need to configure these defaults in order for the 1Password CLI to work. (See Housekeeping Notes, below, for a detailed walk-through.)

For the present, let’s assume I’ve already performed the required bootstrapping and due diligence and that I’m champing at the bit to do something useful in AWS. So I’ll start by creating an IAM role I can use to run AWS Lambda functions. To do this, I’ll use the AWSCLIv2’s aws command:

flox [flox/aws-1pass] daedalus@parousia:~/dev/1pass$ aws iam create-role --role-name LambdaS3ExecutionRole --assume-role-policy-document file://./trust-policy.json

This successfully generates the following JSON:

{
    "Role": {
        "Path": "/",
        "RoleName": "LambdaS3ExecutionRole",
        "RoleId": "AROASBMWSY7QZEGCFNOMJ",
        "Arn": "arn:aws:iam::98127d00a54b:role/LambdaS3ExecutionRole",
        "CreateDate": "2024-07-16T19:49:03+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

In my pre-built Flox 1Password-AWSCLIv2 environment, aws is actually an alias.

You can create, modify, or destroy any Flox environment by making changes to manifest.toml, the configuration artifact stored in the .flox/env/ path nested inside the environment’s project folder. A Flox manifest consists of several sections, including [profile], which is where you define aliases and other shell-specific logic. For my aws-1pass environment, I defined the following logic in profile:

alias aws='op run --session "${OP_SESSION_TOKEN}" --env-file <(printf "%s\n" AWS_ACCESS_KEY_ID=op://${OP_VAULT}/${OP_AWS_CREDENTIALS}/${OP_AWS_USERNAME_FIELD} AWS_SECRET_ACCESS_KEY=op://${OP_VAULT}/${OP_AWS_CREDENTIALS}/${OP_AWS_CREDENTIALS_FIELD}) -- aws'

This alias wraps the aws command, using op run to fetch and inject AWS credentials from 1Password. Because of how op run executes, these secrets are available only for the duration of the command. From my perspective, I’m never prompted to enter my AWS secrets: anything I want to do using aws just works. The same would be true for my teammates (if I had any) because we’d all be using the same pre-built Flox environment. We’ll come back to this in How this works, below.

For now, let me just note that the aws iam create-role command behaved exactly as it’s supposed to.

Doing just about anything in AWS

Now I can go ahead and attach some IAM policies to my lambda-execution-role:

flox [flox/aws-1pass] daedalus@parousia:~/dev/1pass$ aws iam put-role-policy \
    --role-name LambdaS3ExecutionRole \
    --policy-name LambdaS3Policy \
    --policy-document file://lambda_s3_policy.json

With Lambda and other services, I can use my Flox environment's aws alias to do almost anything I can do with the real aws command. If necessary, I can also run the non-aliased aws executable at any time by typing command aws <sub_command> in the CLI.

Say I want to create an AWS Lambda function that watches an S3 bucket and triggers when a file with a .pdf extension gets dropped into it. First I’d build and zip up my Lambda deployment package, just like normal. Then I’d pass this command via my Floxified AWSCLIv2:

aws lambda create-function \
    --function-name PDFProcessor \
    --runtime python3.12 \
    --role arn:aws:iam::98127d00a54b:role/LambdaS3ExecutionRole \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://lambda_function.zip \
    --timeout 15 \
    --memory-size 128

Once again, it works! Or, at least, it’s created. Getting my function to run in Lambda will probably require some work. Still, output like this is a good start:

{
    "FunctionName": "PDFProcessor",
    "FunctionArn": "arn:aws:lambda:us-east-1:98127d00a54b:function:PDFProcessor",
    "Runtime": "python3.12",
    "Role": "arn:aws:iam::98127d00a54b:role/LambdaS3ExecutionRole",
    "Handler": "lambda_function.lambda_handler",
    "CodeSize": 15225283,
    "Description": "",
    "Timeout": 15,
    "MemorySize": 128,
    "LastModified": "2024-07-16T21:22:53.144+0000",
    "CodeSha256": "nAIsdrn73phOjWQAtyvWxMc8yrLqIMdTeI76xa5GaGo=",
    "Version": "$LATEST",
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "a798235a-8e19-4d79-8677-6b196a44cd6c",
    "State": "Pending",
    "StateReason": "The function is being created.",
    "StateReasonCode": "Creating",
    "PackageType": "Zip",
    "Architectures": [
        "x86_64"
    ],
    "EphemeralStorage": {
        "Size": 512
    },
    "SnapStart": {
        "ApplyOn": "None",
        "OptimizationStatus": "Off"
    },
    "RuntimeVersionConfig": {
        "RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:7776dee4c90fff4d4bf833b78d94e5b4148d14d044b867a14756f301ecfe1e28"
    },
    "LoggingConfig": {
        "LogFormat": "Text",
        "LogGroup": "/aws/lambda/PDFProcessor"
    }
}

The JSON-looking text is metadata associated with my Lambda function, which should run, assuming that

• My IAM roles and permissions are correctly configured;
• The code in my lambda_function.zip deployment package runs correctly; and
• This package also includes all required runtime dependencies.

The above example involves Lambda, but you can use the aliased aws command to do basically anything you might want or need to do in AWS. Like create S3 buckets:

flox [flox/aws-1pass] 252 daedalus@parousia:~/dev/1pass$ aws s3api create-bucket --bucket these-names-are-global-so-they-gotta-be-unique --region us-east-1
{
    "Location": "/these-names-are-global-so-they-gotta-be-unique"
}

Or upload files to S3:

flox [flox/aws-1pass] daedalus@parousia:~/dev/1pass$ aws s3 cp /mnt/arkiv/60TBarchive.zip s3://these-names-are-global-so-they-gotta-be-unique/

I kid. I didn’t really upload a 60TB file. (On a side note, does [clearing throat] anyone know if AWS charges for data ingress, as well?) This was just my puckish attempt at showing that anything I can do with the aws command I can do with my alias, too. Well almost anything.

How this works

As I mentioned earlier, the alias in the [profile] section of my Flox aws-1pass1 environment basically functions as a wrapper for aws, the command you use to perform most tasks in AWSCLIv2.

In my testing, the only aws sub-command that I can't pass to it is aws configure, which is a special case. It’s the command the AWSCLIv2 uses to bootstrap its own setup, writing data to a config file that lives in ~/.aws/. Interestingly, aws configure expects to write sensitive data, like your AWS Secret Access Key, to this file. In plain text. Don’t believe me? Check this out:

flox [flox/aws-1pass] daedalus@parousia:~/dev/1pass$ cat ~/.aws/credentials 
[default]
aws_access_key_id = 31415926535897932384
aws_secret_access_key = 3141592653589793238462643383279502884197

I’ve used the digits of pi to obfuscate my AWS credentials because, again, they’re stored in plain text.

By contrast, my Flox environment doesn’t persist sensitive data anywhere. It does export my AWS credentials as environment variables, but it does this in a neat way. First, it takes advantage of the built-in isolation you get with Flox itself: basically, you can access anything on your system when you’re within a Flox environment’s subshell; however, the tools, libraries, and variables inside this environment cannot be accessed from outside of it, even by other Flox environments! Any variables you export while working in a Flox environment are destroyed the moment you exit that environment.

That’s one layer of isolation. But 1Password gives you another, super-neat way to isolate sensitive data. You can use the op run sub-command to spin up tasks in an ephemeral 1Password environment, which is kind of like a virtual environment—or a subshell within a subshell. When you type aws with one or more sub-commands, this transparently invokes op run, which executes the tasks associated with these commands in an ephemeral 1Password environment. It’s kind of like two layers of isolation.

tl;dr: When you use your environment’s aws alias to do something, your AWS secrets exist as env variables only for as long as it takes op run to finish. Moreover, they execute in a subshell inside a subshell.

Housekeeping notes

In our rush to start doing stuff in Lambda and S3, we skipped over what you need to do to bootstrap your 1Password/AWSCLIv2 environment. This is kind of important! So let’s explore this in detail now.

By default, the first time you run op signin from the 1Password CLI, you’ll see the following prompt:

No accounts configured for use with 1Password CLI.
 
You can either:
 - Turn on the 1Password desktop app integration to sign in with the accounts you've added to the app: https://developer.1password.com/docs/cli/app-integration/ for details.
 - Add an account manually with 'op account add' and sign in by entering your password on the command line. See 'op account add --help' for details.
- Authenticate using a 1Password service account by setting the 'OP_SERVICE_ACCOUNT_TOKEN' environment variable to your service account token. Learn more: https://developer.1password.com/docs/service-accounts/ 
 - Use 1Password CLI with a Connect server by setting the 'OP_CONNECT_HOST' and 'OP_CONNECT_TOKEN' environment variables to your Connect host and token, respectively. Learn more: https://developer.1password.com/docs/connect/
 
Do you want to add an account manually now? [Y/n]

This is 1Password’s out-of-the-box setup wizard. You will need to enter the following:

• Your team or organization URL
• Your email address
• Your 1Password secret key

This data is used to generate 1Password’s ~/.config/op/config file. If you’re a user of the 1Password CLI, there’s a good chance this file already exists. Its contents should look something like this:

{
        "latest_signin": "flox_team",
        "device": "<REDACTED>",
        "accounts": [
                { 
                        "shorthand": "flox_team",
                        "accountUUID": "<REDACTED>",
                        "url": "https://<REDACTED>",
                        "email": "<REDACTED>",
                        "accountKey": "<REDACTED>",
                        "userUUID": "<REDACTED>"
                }
        ]
}

If this file doesn’t already exist, you could just copy and customize this JSON, using it to bootstrap ~/.config/op/config yourself. However, with the power of Flox, we can do even better than this!

Using Flox to Bootstrap AWS CLI + 1Password Setup

How much better? As a proof-of-concept, I built an automated setup wizard into my own aws-1pass environment.

This wizard creates and pre-populates a 1Password config file with the required data and offers to set up session persistence. It also gives you the option of obtaining a 1Password session token, so you don’t have to re-authenticate each and every time you activate the environment. Since the token expires after a set period of time, it’s especially useful if you need to activate multiple instances of the same environment, or if your workflow involves exiting (i.e., destroying) and re-entering (i.e., re-launching) the environment.

You can try it out yourself by running flox activate -r barstoolbluz/aws-1pass.

Achieving this involved writing a few hundred lines of shell logic in bash. I could either have included this logic in the [hook] section of my Flox environment’s manifest.toml file or put it into a shell script.

I opted for the former pattern, mainly because I want to be able to remotely activate my aws-1pass environment (using flox activate -r) from FloxHub so I always get the latest version of that environment. Flox environments are completely declarative, so I can edit and version them directly on FloxHub. By managing my Flox environments centrally and activating them remotely, I get exactly the same experience…everywhere. This pattern scales especially well for teams too. When users flox activate -r, they run a locked instance of the environment, so they can’t make changes that might break it.

However, if I were to push my environment to GitHub, GitLab, or AWS CodeCommit, I could instead move this logic into a shell script—say, bootstrap.sh—that lives in the same repo/project folder.

In this pattern, I’d build conditional logic into my Flox environment that checks for the existence of a config file in ~/.config/op/ and—if this file is missing or unpopulated—runs bootstrap.sh. This pattern gives me a way to version and manage my project’s code and runtime dependencies in GitHub, GitLab, or AWS CodeCommit. The downside is that it doesn’t provide the same level of control as when users flox activate -r from FloxHub, because they can make changes to it locally that potentially break it.

Making it your own

So what does this look like in practice? Basically, my shell logic automates three tasks.

1. Bootstrapping the population of the 1Password config file. We covered this above, in Housekeeping Notes. My Flox environment is specific to 1Password, but it’s simple enough to create environments with built-in logic that bootstraps Hashicorp Vault, AWS Secrets Manager, and other secrets-management tools or services, including Google Secret Manager or Azure Vault.

2. Configuring the specific field names used in the 1Password vault. In 1Password, each item can have multiple fields, and these fields can have custom names. To retrieve your AWS secrets programmatically, you need to know the names of these fields.

My setup wizard prompts you for these field names, persisting them in a session file that's used to store information about your Flox 1Password environment.

This file lives in ~/.config/flox and is called 1password.session. Here’s an excerpt:

# BEGIN 1PASSWORD AWSCLI2 BOOTSTRAP CONFIGURATION
export OP_AWS_USERNAME_FIELD="username"
export OP_AWS_CREDENTIALS_FIELD="credential"
# END 1PASSWORD AWSCLI2 BOOTSTRAP CONFIGURATION

The variables OP_AWS_USERNAME_FIELD and OP_AWS_CREDENTIALS_FIELD should map to the corresponding field names in your 1Password vault.

(Note: There is a complete example of the contents of this file at the end of this section.)

3. Configuring persistence for 1Password sessions. This is helpful if you want to exit your environment and re-enter it without logging back in. This is time-limited: the logic built into the [hook] section of my Flox environment’s manifest.toml file (which lives in your project folder’s ./.flox/env/ path) grabs a short-term session token from 1Password. This token usually lives for no longer than 30 minutes.

You can use booleans to turn this on or off in your ~/.config/flox/1password.session file. See below.

If you enable persistence, your tokens are stored in plain-text in 1password.session. This is a short-term token, so it shouldn’t be a problem. But your org might have policies against this.

Now, as promised, here are the contents of a populated 1password.session file in their entirety:

# BEGIN 1PASSWORD AWSCLI2 BOOTSTRAP CONFIGURATION
export OP_AWS_USERNAME_FIELD="username"
export OP_AWS_CREDENTIALS_FIELD="credential"
# END 1PASSWORD AWSCLI2 BOOTSTRAP CONFIGURATION
 
# BEGIN 1PASSWORD SESSION PERSISTENCE CONFIGURATION
ENABLE_1PASSWORD_PERSISTENCE="true"
# END 1PASSWORD SESSION PERSISTENCE CONFIGURATION
 
# BEGIN 1PASSWORD AWSCLI2 CONFIGURATION
OP_VAULT=1password
OP_AWS_CREDENTIALS=awskeyz
# END 1PASSWORD AWSCLI2 CONFIGURATION
 
# BEGIN 1PASSWORD SESSION TOKEN
OP_SESSION_TOKEN=DKyaoqdPpcUq2ZnRCBJYkLYcchv_RuddmpljE6-nAPc
# END 1PASSWORD SESSION TOKEN

Again, my aws-1pass environment is a proof-of-concept: an ambitious example of what you can do with a shared Flox environment. I use it across several different Linux and Windows/WSL2 systems, and I'm productive with it. However, it is also customized for my workflow. Think of it as a framework you can build upon and customize to suit your own needs.

Pragmatic secrets management—for AWS and beyond

Secrets management is supposed to make your life easier, but sometimes the opposite is true!

Flox is your secret weapon.

It’s an environment and package manager with superpowers. You can use it to create reproducible build environments of all kinds, including secure, isolated secrets-management environments that run and behave exactly the same way across dissimilar systems—in collaborative development, when you push them to CI, or in the cloud. You can easily share Flox environments or push them to FloxHub. Pulling Flox environments is exactly as straightforward as cloning repos from GitHub.

The best part is that Flox gives you isolation on your own terms. You get transparent access to everything on your local system because you’re working in an isolated environment on your local system. If you’re missing or need to update tools or libraries, there’s no time-intensive VM or container rebuilding.

Just flox install what you need and push your changes to FloxHub.

Flox is reproducible software builds made simple!

Flox has just a few commands and is easy to learn. This walk-through is a good starting point, and so is our “Flox in Five Minutes” guide. It's free to download and use too. So why not give it a try?