Anton Babenko

How to build, scale, and maintain 45 public Terraform modules with over 100 million provisions by Anton Babenko

Copy and paste of Terraform reduce reusability, maintainability, and scalability of our configuration. In an effort not to repeat ourselves, we might start moving our configuration into modules and run into new scaling and collaboration challenges!

In this talk, I will describe some of the challenges and lessons learned in building, scaling, and maintaining the public Terraform modules for AWS components and how to apply them to your modules.

I defined an initial goal for those modules to provide a powerful and flexible way to manage infrastructure on AWS but with more than a couple thousand issues and pull-requests opened since the beginning, the goal had to change. What started as an initial set of basic Terraform AWS modules for common architectural patterns on AWS soon became a base for many customers using Terraform and required radical changes and improvements.

I will describe some of the challenges along the way and lessons learned when building entirely open-source Terraform modules used by thousands of developers. Some problems are technical such as versioning, quality assurance, documentation, compatibility promises, and upgrading. Other problems are around collaboration and software design principles, such as how to reason about feature-requests or how small should a module be. I will also examine the testing strategy for terraform-aws-modules and discuss the reasoning for not having tests!

I will provide a list of dos and don’ts for Terraform modules that highlight the critical features (e.g., documentation, feature-rich, sane defaults, examples, etc.), which make terraform-aws-modules scalable in collaboration and use.

By the end of the talk, attendees will understand proven practices around building useful, maintainable, and scalable modules, learn about public modules available to them, and how they can participate in making those open-source projects better.

Talk Questions

  • Question 62
    What are your thoughts about Terragrunt? Do you recommend using the tech? Thanks!
  • Question 57
    Mention something you like from Pulumi that does not exist in terraform.
  • Question 63
    When am I supposed to use Ansible vs Terraform ? Terraform sometimes looks like Ansible. Some people says it's complementary.
  • Question 59
    What do you think is the future of terraform and IaC?
  • Question 65
    Would you consider create GCP or Azure modules?
  • Question 61
    If testing is not easy to write, shouldn't efforts be put into making sure writing tests is easier?
  • Question 58
    You claim tests are not that important and I'd agree if TF was purely declarative but the raise of functions makes it more prone to error, somehow. How do we get robustness without tests now? Maybe Localstack?
  • Question 60
    What about testing terrafotm modules using localstack?
  • Question 66
    As tf configurations grow, planning them takes forever. Are there any tips or practices that can help avoiding that?
  • Question 64
    Do you have a test coverage report of some sort?
  • Question 67
    Any strategy to push users to update module versions?
  • Question 68
    Do you think tf may evolve as more than a declarative languaje but to a programming language itself?
  • Question 70
    Why use terraform and not cloudformation or cdk?