Aws on Sander Knape

Terraform 'data source will be read during apply' messages - What is it and how to fix

Sat, 09 Nov 2024 12:43:02 +0200

Terraform users will likely be familiar with “data source will be read during apply” messages that may appear in the plan output. These messages can be confusing and may even lead to unexpected re-creation of resources. Typically, these messages are related to using data sources in combination with Terraform modules and explicit dependencies.

Data sources and modules are two powerful and essential concepts. Data sources allow you to fetch information from existing resources and pass that data to other resources. Modules promote reusability and hiding complexity by encapsulating collections of resources into sharable, versioned packages. Explicit dependencies are also valid to use in various situations. But combining these concepts can lead to confusion and unexpected behavior.

Shift left AWS tag enforcement with Terraform and tfsec

Mon, 03 May 2021 21:03:01 +0200

There are many ways to improve the developer experience of deploying infrastructure into the Cloud. One such method is by shifting left: provide early feedback to shorten the feedback loop and speed up development.

When deploying infrastructure into AWS with an infrastructure as code tool such as Terraform, you can validate that code as part of a CI/CD pipeline. A pull request can automatically receive feedback about the configuration of resources, thus enforcing the environment to stay compliant with the organization’s policies.

A Serverless Payment Workflow using AWS Lambda and the AWS CDK

Tue, 15 Dec 2020 17:38:12 +0200

Serverless technology is getting more popular by the day. More and more people are starting to experiment with it and learn for which use cases it can add value. In this blog post I share an example of what a fully Serverless workflow can achieve.

For a while now I’ve been curious how one would implement a payment workflow on a website. I was aware that platforms like Stripe, Adyen and Mollie exist, but I never knew how much work would be required to set up a fully functioning workflow. I therefore decided to give it a try using nothing but Serverless technology.

Using Amazon Cognito JWTs to authenticate with an Amazon HTTP API

Sun, 02 Aug 2020 16:29:33 +0200

Last year AWS released a new iteration of their API Gateway product: HTTP APIs. This new version promises lower prices, improved performance and some new features. Some features that are available in the older REST API are not (yet) available for HTTP APIs, though. The official comparison page gives a good overview of which features are available in both products.

My favorite new feature available for HTTPs APIs is JWT Authorizers. It is now possible to have the HTTP API validate a JWT coming from an OIDC or OAuth 2.0 provider. While this was already possible using a Lambda Authorizer, now this can be achieved in a fully managed way with only a minimum amount of work required. It’s even easier now to build secure APIs with proper authentication.

Building a static serverless website using S3 and CloudFront

Wed, 12 Feb 2020 19:25:02 +0200

Hosting static websites is great. As they only contain static assets to be downloaded by the visitor’s browser - think HTML, CSS, Javascript, Fonts, images - no server-side code such as Java or PHP needs to be run. They’re therefore typically faster to load than dynamic websites, they have a smaller attack surface, and are easier to cache for even better performance.

That is why some time ago I moved this blog from a Wordpress installation hosted on EC2 to a static website. As I was already in AWS, and I knew that S3 + CloudFront was a popular choice for hosting static websites, I decided to host my blog in S3 with CloudFront in front of it as the CDN.

Roundup of the most important pre-re:Invent 2019 releases - so far

Mon, 25 Nov 2019 11:04:24 +0200

The most exciting time of the year for AWS Enthusiasts is upon us. In exactly seven days, AWS re:Invent 2019 will kick off and everyone is excited to see what great features will be released and announced this time around.

This year especially though, many new features are already released the weeks leading up to re:Invent. If you haven’t been paying attention, it was easy to much some great new announcements. Therefore, in this blog post, a roundup of the (in my opinion) most important AWS releases in the past few weeks.

Five ways to enable developer autonomy in AWS

Tue, 23 Jul 2019 12:00:00 +0200

It hasn’t been that long since it was normal to request compute capacity at some operations department within your organization. In fact, it’s probably still pretty common in some organizations. With the move to virtualization and especially the cloud, this process of course has changed dramatically for the good. Not only compute capacity for applications, but also resources such as databases, queues, load balancers and storage are now available virtually unlimited.

Building serverless applications with the AWS CDK - and testing them locally

Fri, 31 May 2019 16:35:02 +0200

The AWS Cloud Development Kit (AWS CDK) is a new framework for defining Infrastructure as Code (IaC) by AWS. It allows you to write IaC in a set of different languages. At the moment the following languages are supported: Javascript, Typescript, Python, Java, .NET. Support for other languages is coming.

Of course, other methods like CloudFormation and Terraform already exist to write IaC. Using these tools you write declarative code in YAML, JSON or the Hashicorp Configuration Language (HCL) in a mostly declarative state. These tools will also support some basic operators such as if-statements and for-loops. Especially HCL has many of these capabilities with the latest 0.12.0 release. However, it will always be different from using a “real” programming language. And it requires you to use a new tool, instead of using a language you already know.

Blocking account-wide creation of public S3 buckets through a CloudFormation custom resource

Fri, 16 Nov 2018 17:25:02 +0200

Yesterday, AWS announced the release of an important and much-wanted new feature for S3: blocking the creation of public S3 buckets on an account-wide. Enough has been written already about open S3 buckets on the internet. Given that it is very simple to create a public S3 bucket, we regularly learn about new (big) companies that have exposed privacy-sensitive data to the world through such buckets.

The confusion is mainly around opening up your bucket to “everyone”. Where people expect this to mean “everyone in the AWS account”, it actually means “everyone in the world”.

How to easily test your AWS IAM credentials locally

Sun, 02 Sep 2018 18:30:02 +0200

It is still very common to develop an application locally on a laptop/desktop before pushing it to a production-like environment. The local development environment is kept as close as possible to production using technology such as Docker or AWS SAM when working with AWS Lambda. However, when working with AWS resources through Identity and Access Management (IAM) policies, local IAM permissions are typically different from the permissions the application will have in AWS. This inconsistency can cause issues later in the development workflow: an application that fully worked locally can run into errors when ran in AWS if the IAM permission there are different.

Two years with CloudFormation: lessons learned

Thu, 09 Aug 2018 12:43:02 +0200

It’s been close to two years since I started working with AWS CloudFormation, the Infrastructure as Code solution by and for AWS. With CloudFormation you can describe and provision your entire AWS infrastructure in code.

The entire AWS environment I work with is provisioned through CloudFormation - we only have read-only access to our AWS accounts. We now work with a number of different VPCs and many different AWS services. We currently have close to 300 stacks provisioned, describing the foundational networking and over a hundred different applications. All these stacks are deployed in three different accounts (testing, acceptance and production), with CloudFormation giving us the confidence that all these accounts are almost exactly the same (with minor exceptions such as that we run less resources in testing and acceptance to reduce costs).

The hidden challenges of Serverless: from VM to function

Thu, 02 Aug 2018 12:29:02 +0200

Serverless is a relatively new term. It’s a software development paradigm where the entire concept of a “server” is abstracted away from the development process. You essentially only use managed services that handle scaling, and you pay only for what you use. You no longer need to think about operating systems, security patches, scaling configuration and more. All this is handled for you behind the scenes. The most well-known example is AWS Lambda, though as we will see in this blog post, serverless is much more than that.

How to manage any kind of secret with AWS Secrets Manager

Sat, 07 Jul 2018 20:04:02 +0200

AWS Secrets Manager is a service recently released designed to make the management of secrets easier. It provides built-in support for Amazon RDS, making it very easy to set and rotate secrets and use the CLI or an SDK to retrieve secrets from applications. Through the use of custom Lambda functions, essentially any database or an otherwise protected endpoint is supported.

Setting up Secrets Manager for a non-RDS database is less trivial as you need to write your own functionality using AWS Lambda. In this blog post we’ll go through the process of creating a Lambda function for rotating a MongoDB user. First, let’s dive in a bit more into what Secrets Manager can do for us.

Secret management design decisions: theory plus an example

Thu, 08 Mar 2018 22:29:02 +0200

Secret management is one of those security topics that is often an after-thought while designing systems. Passwords are set up manually, shared through non-secure methods such as e-mail or Slack, and password rotation is often neglected because it’s time-consuming and error-prone.

This is a shame because with some effort, secret management can definitely be automated. If done properly, less manual work is required to build and maintain applications, and of course security is increased reducing risk.

Comparing AWS SAM with the Serverless framework

Thu, 22 Feb 2018 20:39:02 +0200

Serverless applications are everywhere these days. Having been introduced some years ago with the introduction of AWS Lambda, today serverless is much more then Function as a Service (FaaS). AWS is even starting to use the term in their products: at AWS re:Invent 2017 “Aurora Serverless” was introduced, a fully managed RDMS database.

How do you build such applications? Given that you properly like the Infrastructure as Code mindset as much as I do, the question is how to properly specify serverless applications provisioned in AWS in code. Two main options are out there: the Serverless Framework and AWS SAM. Both frameworks allow you to make it easier to build serverless applications.

Creating a serverless API using AWS API Gateway and DynamoDB

Sat, 14 Oct 2017 14:54:02 +0200

AWS API Gateway is a managed service from Amazon Web Services that allows you to easily create an API endpoint. An incoming request can either be forwarded to an HTTP endpoint, can invoke a Lambda function, or connect to an AWS service. Arguably, the Lambda integration is most commonly used as it allows you to invoke a piece of code without managing a server or operating system (thus, combined with API Gateway, creating a serverless api). A common use case is to store and retrieve data to/from a database.

A custom CloudFormation resource example for CodeDeploy

Thu, 17 Aug 2017 21:25:02 +0200

CloudFormation is the AWS product for Infrastructure as Code. It allows you to provision AWS resources through a template that describes how to configure that resource. Unfortunately, CloudFormation will sometimes be behind on new features released by AWS. Where the AWS console and API will allow you to deploy resources with a certain configuration, in CloudFormation specific settings might simply not yet be available. If its your goal to deploy your AWS environment completely through Infrastructure as Code, this will block you from doing that. Luckily, through custom resources, CloudFormation allows you to extend the templating language and still give you the option to provision resources entirely through CloudFormation.

Using pre-signed URLs to upload a file to a private S3 bucket

Tue, 08 Aug 2017 18:12:02 +0200

I’ve written about serverless architectures before (a serverless website in S3 and dynamic image generation with Lambda). I strongly believe in this notion as it minimizes required maintenance and makes the initial setup much easier. Of course, there is no silver bullet for every problem but use cases definitely exist where a serverless architecture makes a lot of sense.

One such use case for serverless is a small piece of functionality that you just need to run. Somewhere. You want to care as little as possible about this somewhere because the functionality is so small that pretty much any work you put into the somewhere is more than the time you spend on the actual functionality.

CloudFormation StackSets: automated cross-account/region deployments

Wed, 26 Jul 2017 19:53:02 +0200

Yesterday, AWS released CloudFormation StackSets. A StackSet is a set of CloudFormation stacks that can easily be deployed to multiple AWS accounts and/or multiple AWS regions. Before, each stack had to be deployed separately and custom scripts were required to orchestrate deploying to multiple accounts/regions. Therefore, this feature is bound to make the lives of AWS administrators a bit easier.

There are loads of use cases for deploying stacks to multiple locations. For example, it’s considered a best practice to enable AWS Config in every region. This service keeps track of resources in an AWS account and changes to those resources. AWS Config needs to be enabled in every region separately, so a CloudFormation stack is required for every region.

Building a serverless website in AWS

Mon, 01 May 2017 19:25:02 +0200

Maintaining a web server for your website is not ideal. It’s tedious, takes a lot of time if done properly and distracts from what you really want to do: sell a product, share information, e.t.c. The more time and money you spend on the platform, the less is available to spend on the website and your core business. Unless you’re a hosting company of course, but then this blog post isn’t really for you.

Use Infrastructure as Code for automated security in the deployment pipeline

Mon, 01 May 2017 19:25:02 +0200

Infrastructure as Code (IaC) is a very powerful concept. The idea is that you put all infrastructure resources - networks, subnets, load balancers, firewalls and so on - in code. You then deploy your infrastructure the same way application developers deploy their code: through a continuous integration / continuous deployment (CI/CD) pipeline. Other benefits already reaped by application developers that become available are code linting, automated testing and an audit trail of your changes if combined with a version control system. The most well known IaC tools are Terraform (supports many different services) and CloudFormation (specifically for the AWS cloud).

Dynamic image generation with AWS API Gateway and Lambda

Tue, 21 Feb 2017 20:49:02 +0200

The combination of API Gateway and Lambda is very powerful. It allows you to build some complex functionalities without maintaining any virtual machines yourself. Lambda can be hooked up to many other (AWS) Services including DynamoDB, Kinesis and S3. This paradigm, better known as serverless, is truly upcoming for years now and its certainly time to jump on the bandwagon if you haven’t done so already.

One of the better-known use cases of AWS Lambda is the generation of thumbnail images as images are inserted into S3. In this blog post we will go one step further and generate images based on the URL and return the image as a binary through API Gateway. The number of possibilities this unlocks are pretty much countless, including the generation of charts, bar codes, dynamic icons and much more.

Getting started with AWS Cognito

Sat, 04 Feb 2017 23:32:02 +0200

Did you ever write your own authentication service? In essence it’s quite trivial: allow a user to enter a username and a password. Next, look in a database for a row/document that matches the received data. If found, login. If not, be gone. It becomes harder when you start thinking more about proper security. And what about features such as password resets, login throttling or logins with Google/Facebook? Surely a managed service must exist that can make all this much easier for us.