Keeping Stateful and Stateless Resources in Separate Stacks in CDK
It's a good idea to keep stateful resources like e.g. databases in a separate stack from stateless resources.
When working with the AWS Cloud Development Kit (CDK), one thing I sometimes find quite difficult is to decide how to structure constructs in stacks in a way that allows painless and flexibel deployments on the one hand, but on the other hand also keeps everything maintainable and easily manageable. For example the classic "everything-in-one-stack" approach you sometimes find: Easily manageable from the code perspective because everything is in one place, all references to resources are directly available. But from the deployment view it's a pain, because everything always has to be treated together.
The latter approach becomes especially delicate when there are stateful resources like e.g. S3 buckets or RDS databases together with stateless resources like e.g. Lambdas or SQS queues to be deployed. Sometimes you want to destroy a stack and redeploy it from scratch, be it for cleaning things up after an experiment, be it because something is really badly broken. When there is a stateful resource in the affected stack, this becomes a problem because in most cases you want to keep the data stored in e.g. an RDS database. When destroying a stack, depending on the configured removal policy, the stateful resource gets either
- orphaned (
RemovalPolicy.RETAIN, the default as of today), - deleted (
RemovalPolicy.DESTROY) or - deleted but with a snapshot being saved right before the deletion, so the resource can easily be re-created later (
RemovalPolicy.SNAPSHOT, only available for some resources like for example databases).
Like mentioned, often the deletion is not an option, hence only RemovalPolicy.RETAIN remains which orphans the resource from the stack. The drawback is that - at least from my knowledge - there is currently no way to reintegrate an existing resource into a stack (it seems they are already working on it), which means, the resource is not manageable by CDK anymore when orphaned because it is not part of a stack anymore which is usually also not what you want. Additionally, you might run into naming conflicts when trying to re-deploy the same stack because of the already existing (yet orphaned) resource.
Consider another scenario: You have a construct consisting of different stateful and stateless resources. One day as part of a refactoring it becomes necessary to move this construct to another place or maybe rename it. This leads to resource replacements and results in the same problem as described above.
As a conclusion, from my point of view it is a very good idea to keep stateful resources in separate stacks and also spend sufficient time on thinking about naming and how to structure stateful resources in the code because changing these things later is usually not possible without extra effort. The separation also brings the advantage that you can turn on termination protection on the stateful stack which provides an additional layer of security to ensure that important data is not accidentally deleted. And finally, doing a cdk diff before every non-trivial deployment for showing the impact on the existing infrastructure including potential replacements of resources should be a mandatory step in every deployment workflow.
This post is part of Best Practices, a series of short posts which summarize dos and don'ts I find useful in my daily work. Some of them are common sense, others might be a question of taste, some might even be controversial. All are my personal opinion. You see things differently? Leave me a comment on Mastodon or by E-Mail.