Download

Download

💡

This documents is optimized for mobile view.

Download

🚀 𝗪𝗵𝗲𝗻 𝗱𝗼𝗲𝘀 𝗶𝘁 𝘀𝗵𝗶𝗻𝗲?
• Container start-up configs: inject secrets or hostnames into nginx.conf
• Quick local scripting: stamp today’s date or $HOME into a file in 2 lines
• CI/CD pipelines: render YAML/INI/JSON per-environment without Helm/Jinja

👨‍💻 𝗖𝗼𝗱𝗲:

export IP=$(curl -s https://api.ipify.org)
envsubst '$IP' < hostname.tmpl > hostname.conf

ℹ️ 𝗘𝘅𝗽𝗹𝗮𝗻𝗮𝘁𝗶𝗼𝗻:

1️⃣ curl fetches my public IP address and stores it in the IP environment variable.

2️⃣ envsubst reads the template file, replaces every instance of $IP with that value, and saves the finished file as hostname.conf.

🎞️ Check the 14-second demo below to watch it happen in real time.

TL;DR

• Already dual-stack / IPv6-only? → Use an Egress-Only Internet Gateway (no NAT fees).

• More than 70 % of bytes go to AWS services? → Shift to Gateway/Interface Endpoints and bypass NAT entirely.

• Low-duty or bursty dev/test traffic? → Spin up a Disposable NAT Gateway.

• Are you okay with running and patching EC2? → Use a NAT instance, Fck-NAT AMI, or Spot-based alterNAT.

• Need ≥99.9 % HA with minimal ops? → Stick with a Managed NAT Gateway.

• If none of the above fit, re-evaluate your requirements.

IFTTNAT (IF-This-Then-NAT) Decision Tree

AWS Managed NAT Gateway (Baseline)

Easiest to deploy (fully managed), highly available within an AZ, and scales to high throughput without user intervention​. However, it introduces significant cost at scale (data processing fees) and provides no fine-grained control (no security groups on it, idle costs accumulate)​. Best for teams that value simplicity and uptime over cost, or when traffic levels are low enough that fees aren’t a concern. Common assumption: “It’s AWS-managed so it must be the best choice” – which is true for hassle-free operation, but as we’ve seen, it can be overkill or too costly for some scenarios.

Use Case Diagrams

Public Connectivity:

Access the internet from a private subnet

Private Connectivity:

Access your network using allow-listed IP addresses

The following diagram shows how instances can access on-premises resources through AWS VPN. Traffic from the instances is routed to a virtual private gateway, over the VPN connection, to the customer gateway, and then to the destination in the on-premises network. However, suppose that the destination allows traffic only from a specific IP address range, such as 100.64.1.0/28. This would prevent traffic from these instances from reaching the on-premises network.

The following diagram shows the key components of the configuration for this scenario. The VPC has its original IP address range plus the allowed IP address range. The VPC has a subnet from the allowed IP address range with a private NAT gateway. Traffic from the instances that is destined for the on-premises network is sent to the NAT gateway before being routed to the VPN connection. The on-premises network receives the traffic from the instances with the source IP address of the NAT gateway, which is from the allowed IP address range.

Enable communication between overlapping networks

You can use a private NAT gateway to enable communication between networks even if they have overlapping CIDR ranges. For example, suppose that the instances in VPC A need to access the services provided by the instances in VPC B.

The following diagram shows the key components of the configuration for this scenario. First, your IP management team determines which address ranges can overlap (non-routable address ranges) and which can't (routable address ranges). The IP management team allocates address ranges from the pool of routable address ranges to projects on request.

Each VPC has its original IP address range, which is non-routable, plus the routable IP address range assigned to it by the IP management team. VPC A has a subnet from its routable range with a private NAT gateway. The private NAT gateway gets its IP address from its subnet. VPC B has a subnet from its routable range with an Application Load Balancer. The Application Load Balancer gets its IP addresses from its subnets.

Traffic from an instance in the non-routable subnet of VPC A that is destined for the instances in the non-routable subnet of VPC B is sent through the private NAT gateway and then routed to the transit gateway. The transit gateway sends the traffic to the Application Load Balancer, which routes the traffic to one of the target instances in the non-routable subnet of VPC B. The traffic from the transit gateway to the Application Load Balancer has the source IP address of the private NAT gateway. Therefore, response traffic from the load balancer uses the address of the private NAT gateway as its destination. The response traffic is sent to the transit gateway and then routed to the private NAT gateway, which translates the destination to the instance in the non-routable subnet of VPC A.

Pricing

Links

NAT Gateway Scenarios:

• https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html

Create a NAT gateway:

• https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html#nat-gateway-creating

NAT Instance (DIY)

Essentially free of AWS surcharges – you pay for an EC2 and standard bandwidth, making it much cheaper for high outbound traffic​. You also get more control (security groups, custom AMI possibilities). The trade-off is you manage it: patching, scaling, and handling failover in case of issues​. Suitable for cost-sensitive environments and when you have the expertise to manage the instance. It’s a building block for other solutions like fck-nat and alterNAT. Be ready to address the single-point-of-failure problem, either by accepting short downtimes or implementing your own redundancy.

Diagram

Pricing

Example:

t2.micro Instance + egress fee + EIP costs for 1 hour:

$0.0134/hour + $0.09/GB + $0.005 = $0.1084/h

Link

How to do it yourself:

• https://docs.aws.amazon.com/vpc/latest/userguide/work-with-nat-instances.html

fck-nat (Feasible cost konfigurable NAT: An AWS NAT Instance AMI)

A convenient improvement on the NAT instance approach. You get up-to-date, optimized NAT appliances you can launch with minimal effort. This drastically lowers deployment complexity for NAT instances and ensures you’re not running outdated software. Cost benefits mirror the NAT instance (no per-GB fee, can use tiny cheap instances). The main caution is that it’s a community project; while popular, it’s not an AWS native service, so you assume responsibility for it (albeit a small one). Great for small-to-medium workloads in production, and for development setups where you want to eliminate that ~$30/mo per environment NAT Gateway cost. Think of fck-nat as “NAT Instance 2.0” – easier and safer than rolling your own from scratch.

Pricing

Please refer to the docs for detailed pricing:

https://github.com/AndrewGuenther/fck-nat/blob/main/docs/choosing_an_instance_size.md

Links

• Official Site: https://fck-nat.dev/v1.3.0/

• Deploy with Terraform: https://fck-nat.dev/v1.3.0/deploying/#terraform

• Deploy with CloudFormation: https://fck-nat.dev/v1.3.0/deploying/#cloudformation

• Manual Deployment: https://fck-nat.dev/stable/deploying/#manual-web-console

• GitHub Repo: https://github.com/AndrewGuenther/fck-nat

alterNAT (HA NAT Instance Setup)

A robust solution aimed at larger-scale or mission-critical use of NAT instances. It tackles the reliability concerns by introducing managed failover to NAT Gateways​ and automates maintenance so your NAT instances don’t become pets you forget to update​. It’s ideal for high-volume traffic scenarios (tens of TB per month) where NAT Gateway fees are exorbitant, but you cannot tolerate prolonged outages. The complexity is higher – essentially you’re running a mini service within your infrastructure – but Terraform scripts make it deployable if you’re invested in that ecosystem. alterNAT invites teams to rethink the assumption that “only AWS can provide HA”; it shows you can design a highly available NAT solution yourself​. Use it when you have a clear cost imperative and enough scale to justify the complexity. If your NAT Gateway costs are trivial, stick with simpler options.

Diagram

Pricing

Please refer to to the project docs for the pricing:

https://github.com/chime/terraform-aws-alternat/tree/main?tab=readme-ov-file#background

Links

There are two ways to deploy alterNAT:

• By building a Docker image:

  • https://github.com/chime/terraform-aws-alternat?tab=readme-ov-file#building-and-pushing-the-container-image

• Using Terraform:

  • https://github.com/chime/terraform-aws-alternat?tab=readme-ov-file#use-the-terraform-module

GitHub Repo:

• https://github.com/chime/terraform-aws-alternat

Disposable NAT Gateway

A creative strategy by Shahin Hemmati for very intermittent needs. It’s the ultimate cost saver if you truly don’t need continuous internet connectivity – why pay for NAT 24/7 when you use it 2% of the time? By automating NAT Gateway creation on a schedule, you ensure you “only pay for NAT Gateway while it’s actually needed”​. The obvious limitation is that it doesn’t work for always-online services. It flips the usual assumption (“NAT must be always available”) and asks, what if it isn’t? Use this for controlled environments like nightly maintenance, offline batch processing, or ultra-secure setups where internet access is a gated event. It requires discipline and coordination, but when applicable, it can nearly eliminate NAT costs (and even provide security benefits by closing egress pathways most of the day)​.

Diagram

Pricing

Cost Analysis of the Disposable NAT Gateway Solution:

Cost Analysis of the Disposable NAT Instance Solution:

Links

Deploy using CloudFormation:

• https://github.com/shahinam2/AWS-DevOps-Projects/tree/main/06_Disposable_NAT_Gateway#how-to-deploy-the-solution

GitHub Repo:

• https://github.com/shahinam2/AWS-DevOps-Projects/tree/main/06_Disposable_NAT_Gateway

Egress-Only Internet Gateway (IPv6)

The forward-looking approach. By adopting IPv6, you can avoid NAT altogether for a lot of traffic​. The egress-only IGW is free, high-performing, and simple – if you can use it. This strategy is best when you’re modernizing your network and can ensure both your apps and the services they call support IPv6. It won’t completely replace NAT Gateway until the rest of the world is on IPv6 (and you might still need NAT64 for IPv6-only instances to reach IPv4-only endpoints), but it can dramatically cut down NAT usage. It challenges the common mindset “IPv6 is optional in AWS.” As IPv6 adoption increases, ignoring egress-only gateways could mean literally turning away free bandwidth. Forward-thinking teams in Web, IoT, or cloud-native domains should definitely leverage this: let IPv6 carry as much as possible, and your NAT Gateways (or NAT instances) will only handle legacy IPv4 traffic​.

Diagram

Pricing

There is no charge for an egress-only internet gateway, but there are data transfer charges for EC2 instances that use internet gateways which is ~$0.09/GB depending on the region.

Link

How to add an egress-only internet access to a subnet:

• https://docs.aws.amazon.com/vpc/latest/userguide/egress-only-internet-gateway-working-with.html

Comparison Table

Conclusion

In practice, many organizations will use a combination of these strategies. For example, you might use fck-nat in dev/test, alterNAT in a high-traffic production environment, and enable IPv6 + egress-only IGW to offload part of the traffic, all at once. The AWS ecosystem allows you to mix and match per VPC or per subnet. The key is to evaluate your requirements and usage patterns. If uptime and simplicity trump everything, NAT Gateway is a fine choice – just be aware of the costs (and perhaps monitor them to catch any surprises). If cost is a major concern, explore these alternatives: you might save a lot more than you expect. Just remember that with greater control comes a bit more responsibility. As one engineer noted in a discussion about cutting NAT costs, AWS will always charge you for something; it’s on you to architect smartly around those charges​. In the end, the optimal solution will depend on your team’s tolerance for complexity, your traffic profile, and how much you value a few hundred milliseconds of failover time or a few hundred dollars of savings.

By constructively questioning the default assumptions – “NAT Gateway is the only way” or “managing infrastructure is too risky” – DevOps teams can arrive at a solution that best fits their cost goals and reliability needs. Whether it’s a lean EC2 NAT instance or a full-fledged HA setup with alterNAT, AWS gives us the building blocks to refine our egress architecture. The bottom line: don’t pay $75k in data processing fees if a weekend of engineering can save 40% of that​, and don’t hesitate to use multiple strategies as stepping stones (for instance, using fck-nat now and planning for IPv6 long-term). NAT in AWS is no longer one-size-fits-all – and that’s a good thing for the cloud community’s pocketbook and architecture creativity.

Note:

The above costs are based on the AWS pricing as of April 2025 and based on the Europe (Frankfurt) region.

If you've ever tried creating a custom VPC in AWS, you’ve probably run into the question:
"What CIDR block should I use?"

For many DevOps/Cloud engineers, subnetting feels like a leftover from the old networking world — full of slashes, bit counts, and IP math. But in reality, subnetting is still fundamental when designing resilient, scalable infrastructure in the cloud.

Whether you're allocating public and private subnets, planning for multiple availability zones, or avoiding IP conflicts across environments, CIDR block planning matters.

Thankfully, there's a tool that takes the guesswork out of subnetting: the Visual Subnet Calculator. In this post, I’ll show you how to use it to design your VPC layout in minutes — with no subnetting headaches — and explain how it helps prevent common mistakes cloud engineers make when dealing with CIDRs.

Let’s dive in.

Why Subnetting Still Matters in the Cloud

With the rise of serverless, managed services, and container orchestration, you might think subnetting is an old-school concern — something from the days of rack-mounted switches and spanning tree nightmares.

But that’s a dangerous assumption.

In cloud environments like AWS, subnetting is still the backbone of network segmentation, routing, and security. Here's why it matters:

1. Every VPC Needs a CIDR Block

When you create a Virtual Private Cloud in AWS, you must assign it an IPv4 CIDR block (e.g., 10.0.0.0/16). This CIDR defines your IP address space, and all your subnets must live within it.

Once set, this CIDR becomes the foundation for routing tables, NAT behavior, and internal DNS resolution. There's no way around it — if you miscalculate this upfront, you'll end up re-architecting later.

2. Subnets Define Network Zones

Subnets are not just IP ranges — they’re logical zones. For example:

• Public subnets = internet-facing EC2 instances, Load Balancers

• Private subnets = databases, internal services

• Isolated subnets = services without outbound access

Each of these gets its own CIDR range. That means you have to plan:

• How many IPs are needed per zone

• How many AZs you'll spread across

• Whether you’ll reserve space for future growth

3. IP Conflicts Break Everything

If your subnets overlap — or if your on-prem network overlaps with your cloud CIDR — you’ll run into VPC peering failures, blackholed packets, or broken VPNs.

Careless subnetting is one of the top causes of invisible infrastructure issues in hybrid and multi-cloud environments.

4. Security Depends on Good Boundaries

Security groups and NACLs rely on subnet-level granularity. If your network isn’t segmented properly — say, everything is crammed into a /16 — it becomes much harder to isolate workloads or enforce zero-trust policies.

Proper subnetting allows you to model the principle of least privilege at the network layer.

In Short:

Even in 2025, subnetting is more than a legacy skill — it’s an essential part of responsible cloud infrastructure design.

The problem is, most engineers either:

• Overestimate how much space they need (and waste IPs), or

• Underestimate it and get boxed in

This is where the Visual Subnet Calculator proves to be an invaluable tool. Before diving into how it works, though, we need to understand the fundamentals of designing a VPC and its subnets.

How to Design Your VPC and Subnets (Step by Step)

Before you start clicking around in a subnet calculator, you need to answer a few critical design questions. This ensures your VPC setup is scalable, conflict-free, and aligned with your infrastructure goals.

Follow this checklist step by step — by the end, you’ll know exactly what CIDR to assign to your VPC and how to split it into subnets.

Step 1: Will This VPC Connect to Other VPCs or Networks?

• If Yes: Avoid using overlapping IP ranges.

  • Stick to less common private ranges like 10.1.0.0/16, 172.31.0.0/16, or even 100.64.0.0/10 (for carrier-grade NAT).

  • This avoids conflicts with on-prem networks, peered VPCs, or VPNs.

• If No: You have more freedom. 10.0.0.0/16 is fine for isolated workloads.

Step 2: How Many Environments Do You Need?

Typical examples:

• Dev, Staging, Prod

• QA, Test, Demo

Multiply environments × zones to estimate subnet count.

Example:
If you want 3 environments, each spread across 3 AZs, you’ll need 9 subnets.

Step 3: Do You Need Public & Private Zones?

Many architectures use:

• Public subnets → for Load Balancers, NAT Gateways

• Private subnets → for EC2, RDS, containers

• Optionally isolated subnets → for sensitive services with no outbound access

Multiply again. If you want public + private, you now need:
9 × 2 = 18 subnets total.

Step 4: Estimate the Number of Hosts per Subnet

Use this table to decide your subnet mask (/19, /20, /21, etc.)

AWS VPC subnet size reference (based on a /16 CIDR block)

Quick Guidance:

Example:
If each subnet needs up to 500 containers or EC2s, go with /22 or /21.

Step 5: Choose a VPC CIDR that Can Fit All Your Subnets

Your VPC CIDR needs to be big enough to contain all planned subnets with some room to grow.

Example: You want 20 subnets with /21 ranges.
Each /21 = 2,048 IPs
→ 20 × 2,048 = 40,960 IPs
Choose at least a /16 VPC (which provides 65,536 IPs).

Step 6: Document Your Subnet Layout

Before creating anything in AWS, make a quick plan:

Now you’re ready to plug these ranges into AWS or a subnet calculator.

Steps summary

To make an informed decision about your VPC’s CIDR and subnet layout, follow these six steps:

1. Check for External Connectivity
   Will your VPC connect to on-prem, other VPCs, or VPNs?
   → Avoid overlapping IP ranges.

2. Define Your Environments
   How many environments (e.g., dev, staging, prod) do you need?

3. Decide on Subnet Types
   Will you have public, private, or isolated subnets for each environment?

4. Estimate Hosts per Subnet
   Pick subnet sizes based on expected host counts. Use the /16–/28 table for guidance.

5. Calculate Total IP Need
   Multiply the number of subnets × IPs per subnet → choose a large enough VPC CIDR.

6. Document Your Plan
   Create a subnet table before provisioning in AWS.

Example: Plan for a Multi-AZ, Multi-Env App

Goals:

• Environments: Dev, Staging, Prod

• Availability Zones: 3 AZs

• Subnet Types: Public and Private

• Each subnet should support ~500 IPs

Step-by-step Breakdown:

1. Connectivity: Will connect to on-prem → use 10.1.0.0/16 to avoid overlap.

2. Environments: 3 total (Dev, Staging, Prod)

3. Subnets per AZ:

   • Each env needs 3 public + 3 private = 6 subnets

   • Total across 3 envs: 18 subnets

4. Subnet size:

   • Use /22 → gives 1,022 usable IPs (more than enough)

5. IP count:

   • 18 subnets × 1,022 = 18,396 IPs → /16 (65,534 usable) is sufficient and it has room to grow

6. Document:

Meet the Visual Subnet Calculator

Once you’ve chosen your VPC CIDR block — let’s say 10.0.0.0/16 — Your next step is to divide it into subnets that match your infrastructure needs. This is where most people either:

• Overestimate and waste thousands of IPs per subnet, or

• Underestimate and run out of space when scaling up.

That’s where the Visual Subnet Calculator comes in — a brilliant, no-nonsense tool that mimics exactly how you plan subnets inside the AWS VPC console.

Step-by-step Example

Imagine you’ve created a VPC with the range 10.0.0.0/16. Now you want to split it into smaller blocks — for example, four subnets with around 8,000 usable IPs each.

Try This:

1. Enter 10.0.0.0 and /16 into the Visual Subnet Calculator and click update.

2. Click Update — this shows the full /16 block.

3. Click Divide on the /16 — you’ll get 2 × /17.

4. Click Divide again — each /17 becomes 2 × /18.

5. Keep clicking Divide until you reach /19.

At the /19 level, the calculator shows you 8 subnets, each with:

• 8190 usable hosts

• Ranges like:

  • 10.0.0.0 - 10.0.31.255

  • 10.0.32.0 - 10.0.63.255

  • ...

  • 10.0.224.0 - 10.0.255.255

This is how it’s done in action:

The final result:

You can also undo your divisions!

Why It’s So Useful

• Instantly shows usable IPs per subnet

• Clearly displays start/end IPs

• Lets you experiment interactively with subnet sizes

• Supports joining subnets back together if you go too far

In short, it’s exactly what most engineers need before going into the AWS Console or writing Terraform code.

How Can I Predict This in Advance?

You might be wondering:

“How do I know how many /19s fit inside a /16 before I use a visualizer?”

For that, there’s a helpful tool: the IPTP IP Subnet Calculator

This tool lets you:

• Input a CIDR (like 10.0.0.0/16)

• Choose the desired subnet mask (like /19)

• See the exact number of possible subnets and get full details like usable IPs, broadcast addresses, wildcard masks, and binary subnet masks.

Example:

• Input: 10.0.0.0/16

• Desired subnet: /19

• Result: 8 subnets — from 10.0.0.0/19 to 10.0.224.0/19

• Each has 8190 usable IPs in theory, but remember:

  In AWS VPCs, only 8,187 are actually usable per subnet due to AWS reserving 5 IPs.

(See IPTP calculator screenshots below for full breakdown)

Bonus: Avoiding Common Subnetting Mistakes

Using a visual subnet calculator doesn’t just make planning easier — it helps you avoid critical mistakes that are surprisingly common, especially when working at scale in cloud environments.

Here’s how tools like David C’s Visual Subnet Calculator save you from disaster:

1. Overlapping Subnets

Manually assigning CIDR blocks increases the risk of defining two subnets with overlapping ranges — something AWS won’t allow, but Terraform and cloudformation users can trip over easily.

The calculator shows you visually distinct ranges as you divide subnets, so you can ensure clean, non-overlapping allocations across all Availability Zones and environments.

2. IP Exhaustion

You might think a /24 is enough — until autoscaling kicks in or you add new services.

The tool shows exact usable IPs per subnet, so you can confidently choose between a /23, /22, /21, etc, without guesswork — and leave room for future growth.

3. Poor Network Segmentation

Too often, people just create a /16 VPC and drop everything into a few massive subnets. That kills visibility, auditability, and fine-grained control.

With this tool, you can design subnet boundaries intentionally — separating public-facing components, backend services, database layers, and isolated internal apps using CIDR boundaries that match your architecture.

4. No Room for Expansion

Let’s say you deploy in two AZs today — what happens when your app needs a third?

The visual layout helps you reserve IP space for future AZs or new environments, even if you're not using them yet. This makes your VPC more flexible and future-proof.

Want to Use Visual Subnet Calculator Offline?

Download it here

Or you can even run it with Docker:

git clone https://github.com/davidc/subnets.git
cd subnets
docker build . -t subnets
docker run -d -p 5001:80 --name subnets subnets

Then open http://localhost and start visualizing your subnets without relying on an external site.

Visual subnet calculator github repo

Conclusion

Subnetting doesn’t have to be painful. With the right tools and a bit of planning, you can design scalable, conflict-free VPCs that support any architecture, from simple web apps to multi-environment, multi-AZ cloud platforms.

The Visual Subnet Calculator is a must-have for DevOps engineers, SREs, and cloud architects. It bridges the gap between traditional networking and modern cloud practices, offering:

• Clear visual subnet breakdowns

• Instant feedback on usable IPs

• No guesswork when planning CIDR ranges

Bookmark it, share it with your team, and make it part of your VPC design workflow.

As cloud practitioners, we follow best practices and use multi-account environments. This frequently led to situations where we were cross-referencing resources or viewing logs across multiple accounts. When using the AWS console this becomes quite painful as only one account and region is accessible at a time per browser.

Yes, one way to solve this is to simply stop using the console and develop your own abstractions and visualisation layer on top of AWS’s APIs. However, the native console can be a useful tool for viewing your cloud resources as it provides a user-friendly interface with real-time data, built-in visualizations, and quick access to service-specific dashboards. The AWS console enables intuitive navigation, reducing the need for CLI commands or API calls for simple tasks. It also offers service-specific features like CloudWatch dashboards, cost breakdowns in AWS Billing, and graphical representations of networking components such as VPCs.

TL;DR

Managing multiple AWS accounts efficiently requires selecting the right tool based on workflow preferences and security requirements. Here is an overview of each tool in this post:

• CloudGlance is ideal for those who need a GUI-based solution to manage multiple AWS accounts, organize credentials, and simplify SSH and port forwarding. It supports Firefox Containers for multi-session access and provides Git integration for team collaboration. This makes it particularly useful for DevOps engineers and teams managing complex AWS environments.

• Granted is a CLI-first tool that optimizes role-switching while keeping credentials encrypted. It is designed for users who prefer a fast, terminal-based approach to access multiple AWS accounts securely. While it lacks a GUI and SSH support, it excels in secure authentication and speed.

• Firefox Extensions like Multi-Account Containers offer a lightweight and browser-based way to manage AWS sessions. These extensions help with isolating AWS accounts in different containers, making them a good quick solution for those who mainly work within the AWS Console. However, they lack advanced features like credential management/encryption, SSH, or team collaboration.

• AWS Extend Switch Roles is a browser extension for Chrome and Firefox that enhances AWS’s native role-switching experience. It provides a quick and convenient way to switch between IAM roles inside the AWS Console without needing to re-enter credentials. However, it does not offer multi-session support like CloudGlance or AWS's built-in solution, nor does it handle SSH or team collaboration.

• AWS Built-in Multi-Account Manager, introduced in January 2025, provides a seamless multi-session experience directly within the AWS Console. It allows up to five concurrent sessions across different AWS accounts, eliminating the need for third-party tools for basic account switching. However, it does not support SSH, port forwarding, or external credential management.

Cloud Glance — GUI-Based AWS Account Manager

Intro

Managing multiple AWS accounts across different clients can be a daunting task, especially when switching between environments, accessing private networks, and troubleshooting infrastructure. CloudGlance is designed to simplify this process, providing a unified interface to streamline AWS account management and development workflows.

With CloudGlance, you can:

• Group and visualize all your AWS accounts in a single, intuitive dashboard.

• Easily access AWS Console links, making daily monitoring and troubleshooting more efficient.

• Manage SSH connections by resolving port forwarding conflicts across multiple projects and clients.

For developers and DevOps engineers juggling several AWS accounts, CloudGlance eliminates the hassle of navigating through different environments. It provides quick access to CloudWatch Dashboards, service logs, and other AWS resources—all in one place.

Additionally, when working with SSH, managing VPNs or bastion hosts across multiple AWS environments can lead to port forwarding conflicts. CloudGlance offers visibility into local port usage and their respective forwarding destinations, preventing clashes and simplifying connectivity.

Currently free, CloudGlance may introduce premium options in the future. Stay tuned!

Features​

• CloudGlance manages your ~/.aws/credentials so that you don't have to edit files manually. It's basically a GUI for your ~/.aws files.

• Open multiple AWS consoles at the same time with Firefox Containers.

• CloudGlance creates temporary credentials with STS from either your IAM Role, IAM User, IAM Federated login or AWS SSO as stored in your ~/.aws/credentials. MFA is supported.

• Export temporary STS credentials to your terminal or to another AWS profile.

• Single-click navigation into your most loved AWS service console pages with the help of bookmarks.

• Visualize & manage connections between bastions and available local ports used for port forwarding.

• Both classic SSH(.pem) and AWS SSM are supported for port forwarding.

• Built-in Git support to manage CloudGlance Profiles across your teams. Push, pull and merge JSON configuration profiles at the click of a button.

• The option to encrypt sensitive information in your ~/.aws/credentials like the aws_access_key_id and the aws_secret_access_key without breaking normal AWS CLI commands.

• The Cloud Glance CLI can communicate to the Cloud Glance GUI to obtain temporary credentials and prompt for any user input required, like MFA and SSO.

Installation

Before installing CloudGlance, ensure that you have AWS CLI Version 2 installed and properly configured on your system.

Download AWS CLI Version 2:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Configuring AWS Accounts:

To improve account recognition within CloudGlance, you can assign meaningful names to your AWS accounts in your .aws configuration files. This makes it easier to distinguish between different environments.

Here’s an example of how to define multiple AWS accounts with custom names in your ~/.aws/config file:

Install Firefox and Required Extensions

To ensure CloudGlance works seamlessly, install Mozilla Firefox along with the necessary extensions:

• Firefox Multi-Account Containers - Helps separate AWS accounts into different containers for better organization.

  • https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/

• Open External Links in a Container – Ensures AWS console links open in the correct container.

  • https://addons.mozilla.org/en-US/firefox/addon/open-url-in-container/

Download CloudGlance

CloudGlance is available for Windows, macOS, and Linux.

• https://cloudglance.dev/download

Running CloudGlance on Linux

If you are using Linux, you may need to run the following commands to prevent errors:

chmod +x CloudGlance-0.0.131.AppImage
./CloudGlance-0.0.131.AppImage --no-sandbox

Configuration

In this demo, we will:

• Create a profile in CloudGlance

• Add two AWS IAM accounts to simplify access to their AWS Consoles with a single click

First, create a group:

You can create multiple groups in CloudGlance to better organize and manage your AWS accounts. For example, you might create separate groups for clients, projects, or environments (e.g., Development, Staging, Production).

Once your group is created, open it and start adding AWS IAM profiles. This allows you to quickly access different AWS accounts with a single click, improving efficiency when switching between environments.

Depending on your organization's setup, you need to select the appropriate profile type when adding AWS IAM accounts in CloudGlance. Below is a brief overview of each type:

1. IAM User – Role

• Use Case: You have an existing IAM User who needs to assume an IAM Role (e.g., arn:aws:iam::123456789012:role/YourRoleName).

• Source Profile: Often requires specifying a “source” AWS profile (with user credentials) that can assume the target role.

• Role ARN & External ID: You’ll provide the Role ARN and optionally an external ID if the role trust policy requires it.

• STS Duration: Lets you set how long the temporary session is valid.

2. IAM User

• Use Case: You have a standard IAM User with Access Key and Secret Key.

• Direct Credentials: Stores (or references) those keys for direct usage without an intermediate role.

• MFA Option: If your IAM policies require MFA, CloudGlance can prompt you for a code.

• Federated Login: Typically can generate STS tokens to open the AWS Console or export environment variables.

3. SSO Profile

• Use Case: When using AWS Single Sign-On (IAM Identity Center) to manage user credentials rather than an IAM user and password.

• Browser-Based Flow: You typically authenticate in a browser with your SSO provider (Okta, Azure AD, etc.), and CloudGlance integrates with that.

• Automatic Token Refresh: It retrieves short-lived credentials after you log in through SSO, often with MFA included in the SSO flow.

In this demo, I'll walk you through setting up two personal AWS accounts in CloudGlance using:
IAM User → Federated Login

The first account is called "shahin-new". Below is the configuration needed to add it to CloudGlance:

Now that we have defined our AWS profile in the ~/.aws/config file, let’s add it to CloudGlance for seamless access to the AWS Console.

1. Select "IAM User" – Choose the IAM User tab to configure access using AWS IAM credentials.

2. Select "Federated Login" – This allows login through AWS IAM credentials instead of CLI-only authentication.

3. Enter a Profile Name – This is the display name for the profile in CloudGlance. In this example, we use "shahin-new".

4. Firefox Container Name (Optional) – If using Firefox Multi-Account Containers, set a container name (e.g., "shahin-new"). Keeping it the same as the profile name helps with organization.

5. Choose a Profile Color (Optional) – Assigning colors helps distinguish accounts visually.

6. Select the AWS Profile – Choose the AWS profile you previously configured in the ~/.aws/config file.

7. Access Key ID (Auto-Filled) – This field is automatically retrieved from your AWS credentials file.

8. Secret Access Key (Auto-Filled) – Also auto-filled from your AWS credentials.

9. Region (Auto-Filled) – The default AWS region is automatically retrieved. You can modify this if needed.

10. Assume Policies (Optional) – If your IAM user has permission to assume roles, you can specify any valid IAM policy ARN.

11. In this example, we use:
    arn:aws:iam::aws:policy/AdministratorAccess
    This grants full administrator permissions since the account owner needs unrestricted access.

Follow the same steps to add more AWS accounts, grouping them as needed for easier management.

The result will look like this, allowing you to open the AWS Console with a single click using the displayed button.

Site:

https://cloudglance.dev/

Docs:

https://docs.cloudglance.dev/

Repo:

https://github.com/Systanics/CloudGlance

Granted — CLI-Based AWS Account Manager

Granted is a command line interface (CLI) tool which simplifies access to cloud roles and allows multiple cloud accounts to be opened in your web browser simultaneously. The goals of Granted are:

• Provide a fast experience around finding and assuming roles

• Leverage native browser functionality to allow multiple accounts to be accessed at once

• Encrypt cached credentials to avoid plaintext SSO tokens being saved on disk

Installation

Make sure you have installed Firefox.

Also, make sure that the Granted Firefox extension is also installed:

• https://addons.mozilla.org/en-GB/firefox/addon/granted

AWS CLI v2 should also be installed and it should be configured as shown in the previous section.

Now you are ready to install Granted!

To install it on Windows:

• https://releases.commonfate.io/granted/v0.36.2/granted_0.36.2_windows_x86_64.zip

To install it on Mac:

brew tap common-fate/granted
brew install granted

To install it on Linux:

# install GPG
sudo apt update && sudo apt install gpg

# download the Common Fate Linux GPG key
wget -O- https://apt.releases.commonfate.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/common-fate-linux.gpg

# you can check the fingerprint of the key by running
# gpg --no-default-keyring --keyring /usr/share/keyrings/common-fate-linux.gpg --fingerprint
# the fingerprint of our Linux Releases key is 783A 4D1A 3057 4D2A BED0 49DD DE9D 631D 2D1D C944

# add the Common Fate APT repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/common-fate-linux.gpg] https://apt.releases.commonfate.io stable main" | sudo tee /etc/apt/sources.list.d/common-fate.list

# update your repositories
sudo apt update

# install Granted
sudo apt install granted

# verify your installation
granted -v

Configuration

After installing Granted, open your terminal and run the assume command. Follow the prompts and provide answers based on your requirements or the provided screenshot.

Then, run the assume command again to configure your profile.

If you're using Linux, you'll be prompted to enter a password to create a new keyring. Enter your password and ensure you save it somewhere for future reference.

Finally, to open Firefox and access different accounts in separate isolated tabs:

assume -c <profile-name>
# Examples:
assume -c shahin-new
assume -c shahin-old

You can also use assume -c with the profile selector to quickly choose and open any AWS account you need:

The result will be the same as CloudGlance:

Additional options:

# Opening the console with a specific region
assume -c -r ap-southeast-1
# or
assume -c -r ap-southeast-1 role-a

# Opening the console to a specific service. -s stands for service
assume -s iam

Granted container cleanup

The Granted Firefox extension includes a menu where you can view and clear your tab containers. The menu should appear next to the settings icon as shown below.

Clicking on the icon shows a menu where you can clear your Granted tab containers, as shown below. This is useful if you have roles which you are no longer accessing and you’d like to declutter your tab container list.

Granted Official Website:

https://www.granted.dev

Docs:

https://docs.commonfate.io/granted/introduction

Repo:

https://github.com/common-fate/granted

Using Firefox Extensions — Quick Solution

Container Tab Groups

This extension serves as an alternative to Firefox's official extension, offering additional features that are worth highlighting.

Download page:

• https://addons.mozilla.org/en-US/firefox/addon/container-tab-groups

After installation click the “Get Started” or close the side bar.

There are multiple ways to fully utilize the Container Tab Group extension. One option is the compact sidebar, while another is the panorama grid, an expanded and more user-friendly version that simplifies managing tab groups via containers.

To open the sidebar:

To open the panorama grid:

To manage multiple AWS accounts using this extension, open the sidebar, create as many containers as needed, and name them accordingly. Below is an example of two projects, each containing three accounts.

Alternative:

Download page:

https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/

Using Firefox profiles

A Firefox profile is a user-specific directory where Mozilla Firefox stores personal data, settings, extensions, bookmarks, history, cookies, and preferences. Each profile operates independently, allowing multiple users to have different browsing environments on the same system.

Key Features

• Isolation: Each profile has its own set of data, preventing interference between multiple users.

• Customization: Profiles store user-specific preferences, including extensions, themes, and saved logins.

• Performance Optimization: Creating a fresh profile can help resolve browser issues such as slow performance or crashes.

• Multi-Profile Management: Firefox allows users to create and switch between multiple profiles using the Profile Manager.

Profile Location

The profiles are stored in different locations depending on the operating system:

• Windows:

  C:\Users\<YourUsername>\AppData\Roaming\Mozilla\Firefox\Profiles\

• Linux:

  ~/.mozilla/firefox/

• macOS:

  ~/Library/Application Support/Firefox/Profiles/

Managing Profiles

Open Firefox Profile Manager:

• Run about:profiles in Firefox address bar.

• Click “Create a New Profile”

• Click Next

  

• Choose a proper name, create a folder, choose it as your folder, and click finish

Then you can open, rename, or delete profiles as needed.

You can assign specific profiles for different projects and their stages such as Dev, Staging, or Prod.

You can find more info about Firefox profile manager here:

https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles

AWS Extend Switch Roles

When managing multiple AWS accounts or assuming different IAM roles, constantly switching between them in the AWS Console can be tedious. The AWS Extend Switch Roles extension for Firefox and Chrome makes this process seamless by allowing you to quickly switch roles without manually re-entering credentials.

This browser extension enhances the native AWS role-switching functionality, providing a cleaner UI, color-coded role indicators, and the ability to save multiple role configurations for quick access. In this section, we'll explore how to install, configure, and use AWS Extend Switch Roles effectively to boost productivity in multi-account AWS environments.

Installation

You can install this extension on both Chrome-based browsers and Firefox.

Download Chrome extension:

• https://chromewebstore.google.com/detail/AWS%20Extend%20Switch%20Roles/jpmkfafbacpgapdghgdpembnojdlgkdl

Download Firefox extension:

• https://addons.mozilla.org/en-US/firefox/addon/aws-extend-switch-roles3

Configuration

After installation, left-click on the extension logo and navigate to the configuration section of the extension.

Enter your Role ARN and Region, configure the extension in INI format, and press Save.

Go to the account from which you want to switch roles, and the extension will begin functioning.

From this point, you can switch between as many roles as needed.

AWS Builtin Multi-Session Support — New Feature

On 16 Jan 2025 AWS announced multi-session support, which enables AWS customers to access multiple AWS accounts simultaneously in the AWS Console. AWS Customers can sign-in to up to 5 sessions in a single browser, and this can be any combination of root, IAM, or federated roles in different accounts or in the same account.

When you enable multi-session support, the console URL contains a subdomain (for example, https://000000000000-aaaaaaaa.us-east-1.console.aws.amazon.com/console/home?region=us-east-1). Be sure to update your bookmarks and console links.

You must opt-in to multi-session support by choosing Turn on multi-session in the account menu in the AWS Management Console, or by choosing Enable multi-session on https://console.aws.amazon.com/. You can opt-out of multi-sessions at any time by choosing Disable multi-session on https://console.aws.amazon.com/ or by clearing your browser cookies. Opt-in is browser-specific.

With add session, you can add up to 5 different accounts.

Conclusion

Table of Comparison

Final Thoughts

• CloudGlance is best for teams and DevOps professionals managing multiple AWS accounts with SSH access and port forwarding needs.

• Granted is perfect for security-conscious CLI users who need quick and encrypted role-switching.

• Firefox Extensions offer a simple, container-based solution for isolating AWS sessions, best suited for browser-heavy workflows.

• AWS Extend Switch Roles is a convenient, browser-based IAM role-switcher, best for users frequently switching roles inside AWS Console.

• AWS Built-in Multi-Account Manager is the easiest way to manage multiple AWS accounts within AWS Console but lacks flexibility beyond basic session handling.

Each tool has its own strengths, and the choice depends on whether you prioritize GUI, CLI, multi-session handling, SSH support, or security.

DeepSeek-R1 is an advanced open-source artificial intelligence model developed by the Chinese startup DeepSeek. It is designed to excel in complex reasoning tasks, including mathematics, coding, and logical problem-solving. Notably, DeepSeek-R1 achieves performance comparable to leading models like OpenAI's o1, but with significantly lower development costs and computational requirements.

Significance of DeepSeek-R1:

• Cost Efficiency: Developed with a budget of less than $6 million, DeepSeek-R1 challenges the high-cost approaches of competitors, making advanced AI more accessible.

• Open-Source Accessibility: By open-sourcing DeepSeek-R1, DeepSeek promotes transparency and collaboration, allowing researchers and developers worldwide to study, modify, and enhance the model.

• Technological Impact: The model's emergence has prompted a reevaluation of AI development strategies, emphasizing efficiency and innovation over sheer computational power.

Advantages of Running DeepSeek-R1 Locally:

• Data Privacy: Processing data on local machines ensures that sensitive information remains secure, mitigating risks associated with transmitting data to external servers.

• Customization: Running the model locally allows for tailored modifications to meet specific project requirements, facilitating experimentation and optimization.

• Reduced Latency: Local deployment eliminates the need for internet-based API calls, resulting in faster response times crucial for real-time applications.

• Cost Savings: Operating the model on local hardware can reduce expenses related to cloud-based services and data transfer.

Key Considerations for Running DeepSeek-R1 Locally

Before proceeding, keep the following DeepSeek-R1 models and their corresponding sizes in mind:

When running DeepSeek-R1 locally on your computer, you should consider the following factors:

1. Hardware Requirements

• VRAM (GPU Memory):

  • The different model sizes range from 1.1GB (1.5B model) to 404GB (671B model).

  • If you have a consumer-grade GPU (e.g., RTX 3060, 3070, 4080), you should opt for the 7B model (4.7GB VRAM required) or the 8B model (4.9GB VRAM required).

  • Larger models (14B, 32B, 70B) require more powerful GPUs with at least 10GB+ VRAM.

• CPU and RAM:

  • A powerful CPU (e.g., AMD Ryzen 9 / Intel i9) is recommended for inference.

  • You will need at least double the VRAM size in system RAM. For example, if you run the 7B model (4.7GB VRAM), you should have at least 16GB RAM for smooth performance.

• Storage:

  • Ensure you have enough disk space. The 7B model alone requires 4.7GB, while the larger models (70B+) need hundreds of gigabytes.

  • An NVMe SSD is preferable for faster model loading.

2. Software Requirements

• CUDA / ROCm (for GPU Acceleration)

  • If you have an NVIDIA GPU, install the latest CUDA and cuDNN.

    • When you install the NVIDIA GeForce driver, it typically includes the CUDA runtime libraries—the components needed to run CUDA-accelerated applications.

  • For AMD GPUs, you will need ROCm.

  • If you use CPU-only inference, performance will be significantly slower.

3. Model Selection

• Choose a model that balances performance and hardware limitations:

  • 1.5B: Very lightweight, suitable for older GPUs or CPU-only.

  • 7B / 8B: Good for mid-range GPUs with 6GB+ VRAM.

  • 14B+: Requires high-end GPUs (e.g., RTX 3090, A100, H100).

4. Optimization & Performance

• Quantization: Reducing precision (e.g., GGUF 4-bit, 8-bit quantization) helps reduce VRAM usage.

• Batch Size / Context Length: Adjust to balance response quality and speed.

• Multi-GPU: If you have multiple GPUs, some inference frameworks support model sharding.

Ollama and Deekseek Installation

Before starting with the installation process make sure that if you are using windows, your nvidia graphic card is up to date. you can download and install the latest version of the graphic card from here: https://www.nvidia.com/en-us/geforce/drivers/

To find out whether Ollama supports your GPU you can visit: https://github.com/ollama/ollama/blob/main/docs/gpu.md

First, install Ollama and let it run in the background:

• For Windows: https://ollama.com/download/windows

• For macOS: https://ollama.com/download/mac

• For Linux:

curl -fsSL https://ollama.com/install.sh | sh

Next, download and install the model version that best fits your needs based on the explanation above.

To do this:

1. Visit Ollama’s DeepSeek-R1 Library: https://ollama.com/library/deepseek-r1

2. Choose your preferred model version (e.g., 8B).

3. Copy the provided command and paste it into your terminal.

This installation process is the same for Windows, macOS, and Linux.

At this stage, you can start using DeepSeek-R1 directly from the command line. However, to create a more ChatGPT-like experience, we will install AnythingLLM for an enhanced user interface.

AnythingLLM Installation and Configuration

• Visit AnythingLLM Desktop: https://anythingllm.com/desktop

• Download and install the appropriate version for Windows, Linux, or macOS.

Configure AnythingLLM:

• Open AnythingLLM after installation.

• Follow the configuration steps as shown in the screenshots below to set it up properly.

This setup will enhance your experience by providing a ChatGPT-like interface for interacting with DeepSeek-R1 locally. 🚀

Go to settings:

Configure your LLM provider and then go back to its main page:

Create a new workspace, give it a name, and save it:

Go to the settings of your workspace and configure it according to the screenshot:

After that choose default or new thread to start a new conversation:

Congratulations! You now have a powerful OpenAI-O1-like model running locally on your machine! 🚀

AnythingLLM Alternative

If you're looking for an alternative to AnythingLLM, you can also use LM Studio. One key advantage of LM Studio is that you can install models directly from within the app, eliminating the need for manual downloads or additional setup.

How to Get Started with LM Studio

1. Install LM Studio – Download and install the software from https://lmstudio.ai/

2. Search for Your Model – Use the built-in search feature to find DeepSeek R1 or any other model.

3. Install & Run – Click to install the model directly from the app and start chatting instantly.

This makes LM Studio a convenient and user-friendly option for running local AI models with minimal hassle. 🚀

Then start asking questions:

Ollama commands

To see the version of Ollama installed on your system:

ollama -v

To see a list of installed models with Ollama:

ollama list

To see how Deepseek is performing on your system:

# First run deepseek in teminal:
ollama run deepseek-r1:8b --verbose
# To exit chat mode:
/bye

Ask a question and check the stats at the end.

To determine if the model is running on your CPU or GPU, use the following command:

ollama ps

Output:

NAME              ID              SIZE      PROCESSOR          UNTIL
deepseek-r1:8b    28f8fd6cdc67    6.3 GB    26%/74% CPU/GPU    28 seconds from now

To make sure that your system has detected your GPU, you can check the server.log located at C:\Users\<username>\AppData\Local\Ollama

Feel free to drop any questions in the comments section—I’m happy to help! 😊

Accidentally committing sensitive information, such as API keys, passwords, or personal data like phone numbers, to a Git-based version control system can happen to anyone.

Now, imagine a situation where you’ve pushed an API key that cannot be regenerated, a password that cannot be reset, or—worse—a personal phone number that’s now publicly accessible. No one wants to deal with the hassle of purchasing a new SIM card or facing potential security risks simply because of an oversight.

Fortunately, there are effective ways to address this issue and prevent sensitive information from being permanently exposed.

Important Note Before Proceeding:

⚠️ The following solutions involve rewriting commit history, which will modify all commit hashes. If your workflow depends on commit hashes, consider alternative approaches.

⚠️ For teams, rewriting history can impact uncommitted changes made by your teammates. Ensure you coordinate with your team before proceeding.

Solution 1: BFG Repo-Cleaner

The BFG Repo-Cleaner is a powerful Java-based tool that simplifies the process of removing sensitive information from your Git repository’s history. Below are two methods for setting it up:

The Automated Method

To streamline the setup, use the following script, which has been tested on Ubuntu 24.04:

# The following script has been written and tested on ubuntu 24.04:
curl -s https://raw.githubusercontent.com/shahinam2/bfg-repo-cleaner-auto-install/main/auto-install.sh -o auto-install.sh && chmod +x auto-install.sh && ./auto-install.sh

You can find the repository for this script at: GitHub Repo.

The Manual Method

1. Download the BFG Repo-Cleaner:

   • Visit the official website: BFG Repo-Cleaner.

2. Install Java:

   • Download and install Java (version 8 or later).

This method provides more control over the installation process and is ideal if you prefer a manual setup.

How to use BFG Repo-Cleaner

⚠️ Important: Before proceeding, ensure you have backed up your repository by cloning it into a separate directory. This will help prevent accidental data loss during the cleanup process.

Suppose your repository contains a file named credentials that holds sensitive information, and this file was committed in commit number 2.

Remove the File

First, delete the file containing sensitive information from your local directory:

rm credentials

Stage and Commit the Deletion

Next, stage the change and create a new commit locally to remove the file:

git add .
git commit -m "remove the credentials file"

This ensures that the sensitive file is no longer part of your working directory or future commits.

Using BFG Repo-Cleaner

Assuming you are already in the directory where BFG Repo-Cleaner is located, use the following command to rewrite the repository history and remove the sensitive file:

java -jar bfg-1.14.0.jar /path/to/your/repo/ --delete-files file-to-remove

Example:

If the file to be removed is named credentials, the command would look like this:

java -jar bfg-1.14.0.jar /home/shahin/bfg-tool-test/ --delete-files credentials

Output:

Using repo : /home/shahin/bfg-tool-test/.git

Found 2 objects to protect
Found 3 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/main

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 7b439b14 (protected by 'HEAD')

Cleaning
--------

Found 4 commits
Cleaning commits:       100% (4/4)
Cleaning commits completed in 30 ms.

Updating 2 Refs
---------------

        Ref                        Before     After   
        ----------------------------------------------
        refs/heads/main          | 7b439b14 | 1597a9c1
        refs/remotes/origin/main | d611e7c6 | e063add1

Updating references:    100% (2/2)
...Ref update completed in 38 ms.

Commit Tree-Dirt History
------------------------

        Earliest      Latest
        |                  |
          .    D    D    m  

        D = dirty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After   
        -------------------------------------------
        First modified commit | ab6b164d | a11e11a6
        Last dirty commit     | d611e7c6 | e063add1

Deleted files
-------------

        Filename      Git id          
        ------------------------------
        credentials | df20d103 (52 B )


In total, 5 object ids were changed. Full details are logged here:

        /home/shahin/bfg-tool-test.bfg-report/2025-01-11/23-01-56

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

What the Output Means:

Protected Commits:

• BFG preserved the commit 1bd6a8cb because it was "protected by 'HEAD'." Protected commits are typically the latest commits in your repository to ensure no accidental data corruption.

• If the credentials file exists in this protected commit, you need to clean it manually before running BFG again.

Cleaning Results:

• BFG identified and cleaned 4 commits in your repository that had the credentials file in their history.

• It updated 2 references (refs/heads/main and refs/remotes/origin/main) to point to the rewritten history.

Commit Tree-Dirt History:

• Indicates which commits were modified or cleaned. D (dirty commits) were fixed by BFG.

Deleted Files:

• BFG successfully found and flagged the credentials file (df20d103 is its Git ID). This indicates the file was removed from the Git history for the rewritten commits.

Steps to Finalize and Verify

Run Garbage Collection:

After running BFG Repo-Cleaner, the process is not fully complete. To finalize the cleanup, you must remove any deleted objects from the Git repository by executing the garbage collection command provided in the output.

Run the following command to clean up your repository:

# Ensure you are in the root directory of your repository before executing this command
git reflog expire --expire=now --all && git gc --prune=now --aggressive

Expected output:

Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 612 bytes | 612.00 KiB/s, done.
Total 7 (delta 2), reused 7 (delta 2), pack-reused 0
remote: Resolving deltas: 100% (2/2), done.
To github.com:shahinam2/bfg-tool-test.git
 + d611e7c...1597a9c main -> main (forced update)

This will permanently delete the orphaned objects (e.g., the credentials file) from the repository.

Verify Removal of Sensitive File:

Ensure that the sensitive file, such as credentials, has been completely removed from your Git history by running the following commands:

git log --all --grep="credentials"
git grep "credentials"

If both commands return no results, it means the file has been successfully removed after running BFG and performing garbage collection.

Force Push to Remote: If your repository is hosted on a remote platform (e.g., GitHub), you’ll need to push the rewritten history using a force push:

git push origin --force

⚠️ Important: Force-pushing will overwrite the repository history on the remote server. Be sure to notify all collaborators, as they will need to re-clone the repository to avoid conflicts.

Before Cleanup:

• The commit history shows commit 2, which contains the sensitive credentials file.

• The file's content is visible within the repository.

After Cleanup:

• commit 2 has been rewritten to remove the sensitive content using BFG Repo-Cleaner.

• The sensitive credentials file is no longer visible in the repository history, as confirmed by the updated commit structure.

Solution 2: Git-Filter-Repo

git-filter-repo offers several advantages over BFG Repo-Cleaner, making it a more versatile and efficient choice for rewriting Git history:

1. Flexibility:

   Provides comprehensive features for history rewriting, unlike BFG, which focuses on specific tasks like removing sensitive data.

2. No Java Dependency:

   Python-based and lighter on resources, whereas BFG requires Java Runtime Environment.

3. No Protected Commits:

   Can modify all commits, including the latest, ensuring complete data removal. BFG protects the latest commit by default, requiring manual cleanup.

4. Speed:

   Optimized for performance and generally faster for large repositories compared to BFG.

5. Active Maintenance:

   Regularly updated with detailed documentation, ensuring compatibility with modern Git features.

6. Customizable Outputs:

   Generates detailed logs and mappings for auditing, unlike BFG’s simpler reporting.

7. Core Git Integration:

   Relies on core Git functionality, making it portable and easier to integrate into workflows.

8. Broad Use Cases:

   Versatile enough for various history-rewriting tasks, not limited to removing sensitive data.

How to Use Git-Filter-Repo

Install git-filter-repo:

If not already installed, use the following command:

pip install git-filter-repo

Backup Your Repository:

Before making changes, back up your repository to prevent data loss:

cp -r /path/to/your/repo /path/to/your/repo-backup

# Example:
cp -r /home/shahin/bfg-tool-test /home/shahin/bfg-tool-test-backup

Remove the File:

Run the following command to remove all instances of the credentials file from your Git history:

# Assuming that the credential file is in the current folder, otherwise provide the full path
git filter-repo --sensitive-data-removal --invert-paths --path credentials

-path credentials: Targets the credentials file.

-invert-paths: Removes the targeted file from the repository's history.

This ensures the sensitive file is completely removed from the repository while keeping other data intact.

Force-Push the Updated Repository:

If your repository is hosted on a remote platform (e.g., GitHub), you need to push the rewritten history with a force push:

git push origin --force

Verify Removal:

Run these commands to confirm the credentials file is no longer in the repository history:

git log --all --grep="credentials"
git grep "credentials"

No results indicate successful removal.

Before Cleanup: Commit history shows commit 2, which contains the sensitive credentials file, with its content visible in the repository.

After Cleanup:

After running the cleanup steps, commit 2 has been successfully removed from the history, ensuring the sensitive content is no longer accessible.

Git Guardian: Prevention Over Cure

Avoid the hassle of removing sensitive data from your repository by preventing it from being pushed in the first place. Tools like Git Guardian can monitor and block sensitive information from being added to Git-based platforms.

Visit Git Guardian: https://www.gitguardian.com/

Sources

BFG Repo-Cleaner

• GitHub Repository

• YouTube Tutorial

Git-Filter-Repo

• GitHub Repository

• YouTube Tutorial

• GitHub Guide

Photo by Katarina Humajova on Unsplash.

The Issue

The reason why you have landed here is most probably because of the following error:

Solution

First, create a gpg key:

gpg --generate-key

Expected output after answering all questions:

Then run pass init with the string shown in the screenshot.

Now the sign-in button works as expected.

Sources:

https://docs.docker.com/desktop/get-started/#credentials-management-for-linux-users

https://www.youtube.com/watch?v=AMcvwqvgU5U&t=590s

In recent years, platforms like LinkedIn have been rife with debates about the value of IT certifications. Many senior professionals dismiss certifications as worthless. But how accurate is this claim? Let’s take a closer look at both sides of the argument and unpack why IT certifications can still hold significant value—if approached correctly.

Why the Critics Are Wrong

To illustrate the value of IT certifications, let me share a personal experience from my career.

In 2021, I transitioned from being a network engineer to a DevOps professional. At the time, my Linux knowledge was minimal—essentially limited to using the ls command. Realizing the importance of Linux in my new role, I decided to approach my learning journey methodically. Instead of relying on random YouTube videos or scattered online tutorials, I opted to pursue the CompTIA Linux+ certification. This decision added structure and focus to my learning process.

A Strategic Approach to Learning

The first step was selecting the right learning materials. After some research, I found ITProTV, where Don Pezet’s teaching style resonated perfectly with me. His lessons provided clarity, making complex topics approachable.

While studying, I built my own lab environment using VMware Workstation and multiple Linux distributions. This hands-on approach allowed me to experiment, break systems, and troubleshoot issues—creating a dynamic “playground” for learning. Over the course of three months, I developed a deep understanding of Linux concepts by repeatedly applying what I learned.

Reinforcing Knowledge Through Practice

Once I completed the study material, I began preparing for the Linux+ exam with practice tests. For the questions I didn’t fully understand, I recreated the scenarios in my lab environment to grasp the underlying problems and solutions. This targeted approach not only prepared me for the exam but also reinforced my practical skills.

By the time I passed the Linux+ exam on my first attempt, I had built a solid foundation of Linux knowledge—one that would later prove invaluable in real-world scenarios.

Career Impact

Shortly after earning my Linux+ certification, I landed a job as a 2nd Level IT Support Engineer (DevOps Engineer). My manager explicitly stated that my Linux+ certification was a key factor in their hiring decision, as it demonstrated both my technical competence and my commitment to self-improvement.

While the challenges I faced at work often extended beyond what I had learned, the strong foundation I built enabled me to research and resolve complex issues effectively. This underscores the value of certifications when pursued correctly.

When the Critics Are Right

Unfortunately, not everyone approaches certifications with the same dedication. I’ve encountered individuals who, despite having no prior cloud experience, quickly amassed multiple AWS and Azure certifications by relying solely on exam dumps. By memorizing answers without gaining practical knowledge, they managed to pass exams but lacked the skills required to perform in real-world environments.

The result? They often fail interviews and disappoint hiring managers, who then take to platforms like LinkedIn to denounce the value of IT certifications. These instances reflect a misuse of the certification process rather than an inherent flaw in certifications themselves.

The Bottom Line

IT certifications are only as valuable as the effort and approach you put into earning them. If you:

• Select comprehensive learning resources,

• Dedicate time to hands-on practice and experimentation,

• Use practice exams to bridge gaps in your knowledge,

then your certification can become a powerful asset. It can open doors to new career opportunities and provide a strong foundation for mastering advanced technologies.

However, if you shortcut the process by relying solely on exam dumps and neglecting practical experience, your certification will hold little to no value.

Final Thoughts

When pursued with integrity and dedication, IT certifications remain a worthy investment. They not only validate your skills but also facilitate continued learning and career growth. For those willing to put in the effort, certifications can still be a game-changer.

The Software Development Life Cycle (SDLC) is a structured process used for developing software systems. It defines a series of steps or phases to guide teams in planning, creating, testing, deploying, and maintaining software. SDLC helps ensure that the software is high-quality, meets user requirements, and is delivered within budget and on time.

Waterfall Model

The Waterfall Model is one of the earliest and simplest SDLC models, following a sequential approach where each phase must be completed before the next phase begins. It is linear and structured, making it easy to follow for smaller projects with well-defined requirements.

Phases in the Waterfall Model

1. Requirement Analysis:

   • Gather and document the requirements for the system.

   • Requirements are clear and unchanging.

2. System Design:

   • Translate the requirements into a detailed system architecture.

   • Define hardware and software specifications.

3. Implementation (Coding):

   • Develop the software based on the design specifications.

   • Testing for each module is conducted at this stage.

4. Integration and Testing:

   • Combine all modules and test the entire system.

   • Identify and fix defects.

5. Deployment:

   • Deliver the software to the customer or make it live in the production environment.

6. Maintenance:

   • Handle updates, fixes, and enhancements based on user feedback.

Agile Model

The Agile Model is an iterative and incremental software development methodology that emphasizes flexibility, collaboration, and customer involvement. Instead of completing the entire project in one go, Agile delivers the software in small, manageable parts called iterations or sprints, each lasting 1-4 weeks. At the end of each iteration, a functional and usable product increment is delivered.

Agile is designed to adapt to changing requirements, foster collaboration between cross-functional teams, and prioritize customer satisfaction by delivering high-quality software quickly and continuously.

Phases of the Agile Model

1. Concept or Planning:

   • Define high-level goals, vision, and a broad roadmap.

   • Identify key stakeholders and requirements.

2. Iteration Planning:

   • Plan the objectives and deliverables for the upcoming sprint.

   • Define a prioritized list of features or user stories (often tracked in a backlog).

3. Design and Development:

   • Develop small parts of the software during the sprint.

   • Use collaboration tools (e.g., Jira, Trello) to track progress.

4. Testing and Feedback:

   • Test the product increment at the end of each sprint.

   • Gather feedback from stakeholders or users to refine the product.

5. Release:

   • Deliver working software at the end of each sprint.

   • The release may go live (for the user) or stay in-house for further refinement.

6. Review and Retrospective:

   • Review the sprint's achievements and identify areas for improvement.

   • Plan the next sprint based on lessons learned.

DevOps Model

The DevOps Model is a software development methodology that emphasizes collaboration between development (Dev) and operations (Ops) teams. The goal is to break down silos between these teams to enable continuous integration, delivery, and deployment of software. It focuses on automation, faster delivery, and improving the reliability of software systems.

The DevOps Model combines cultural philosophies, practices, and tools to accelerate the software delivery lifecycle while maintaining high quality, stability, and security.

Phases of the DevOps Lifecycle

1. Plan:

   • Define the product vision, requirements, and roadmap.

   • Use tools like Jira or Trello for project management.

2. Develop:

   • Write, build, and test code continuously.

   • Leverage CI tools like Jenkins, GitHub Actions, or GitLab CI.

3. Build:

   • Automate the creation of builds using tools like Maven, Gradle, or npm.

4. Test:

   • Run automated tests to verify the code's functionality and quality.

   • Tools like Selenium, TestNG, or JUnit are commonly used.

5. Release:

   • Deploy the code to staging or production environments.

   • Tools like Ansible, Chef, and Terraform enable automated deployments.

6. Deploy:

   • Continuously deliver updates to end-users without downtime.

   • Use tools like Kubernetes, Docker, and AWS ECS for container orchestration.

7. Operate:

   • Monitor the system's performance and reliability in production.

   • Tools like Prometheus, Grafana, and Datadog are commonly used.

8. Monitor:

   • Track system health, application performance, and user feedback.

   • Use alerts and dashboards to ensure system reliability.

Key Principles of the DevOps Model

1. Collaboration and Communication:

   • Promotes a shared responsibility culture between development and operations teams.

   • Encourages open communication and shared objectives.

2. Automation:

   • Automates repetitive tasks like testing, integration, deployment, and monitoring.

   • Uses tools like Jenkins, Ansible, Docker, Kubernetes, and Terraform.

3. Continuous Integration (CI):

   • Developers frequently integrate code changes into a shared repository.

   • Automated tests verify the quality of the code.

4. Continuous Delivery (CD):

   • Ensures that software is always in a deployable state.

   • Automates the process of delivering changes to staging or production environments.

5. Infrastructure as Code (IaC):

   • Manages infrastructure (e.g., servers, networks) using code.

   • Tools like Terraform and AWS CloudFormation make infrastructure provisioning reliable and reproducible.

6. Monitoring and Feedback:

   • Uses tools like Prometheus, Grafana, and Splunk for real-time monitoring.

   • Feedback loops help identify and resolve issues quickly.

Iterative Model

The Iterative Model is a software development approach in which a project is developed and refined through repeated cycles (iterations). Instead of delivering the final product in one go, the development process starts with a basic implementation of the requirements and gradually improves it with each iteration. Every iteration involves the stages of planning, designing, coding, and testing, resulting in an incrementally improved version of the software.

The model is particularly useful for projects where requirements are not fully understood at the beginning and need to evolve over time.

Phases of the Iterative Model

1. Initial Planning:

   • Define the high-level goals and scope of the project.

   • Identify core requirements that need to be addressed in the first iteration.

2. Design:

   • Create a design for the current iteration based on defined requirements.

   • Focus on modularity to allow for incremental additions.

3. Implementation:

   • Develop the system for the current iteration, focusing on adding specific features or functionality.

4. Testing:

   • Test the software for defects and validate functionality.

   • Incorporate feedback to refine the product.

5. Evaluation:

   • Present the iteration to users or stakeholders for feedback.

   • Identify areas for improvement and plan the next iteration.

6. Repeat:

   • Start the next iteration, incorporating feedback and new requirements.

Spiral Model

The Spiral Model is a risk-driven software development process that combines the iterative nature of the Iterative Model with the systematic aspects of the Waterfall Model. It emphasizes risk assessment and mitigation in each phase, making it particularly suitable for large, complex, or high-risk projects. The model is visualized as a spiral with multiple loops, each representing a phase of development.

Each loop in the spiral includes planning, risk analysis, development, and evaluation, allowing for incremental improvements with a strong focus on identifying and addressing risks.

Phases of the Spiral Model

Each loop in the spiral consists of four main quadrants:

1. Planning:

   • Identify objectives, constraints, and alternatives for the project.

   • Deliverable: Initial requirements and plans for the current phase.

2. Risk Analysis:

   • Assess potential risks and develop strategies to mitigate them.

   • Prototyping is often used to address uncertainties.

   • Deliverable: Risk management plan or prototype.

3. Development and Validation:

   • Design, code, and test the software for the current iteration.

   • Deliverable: A functional version of the product (or part of it).

4. Evaluation:

   • Present the iteration to stakeholders for feedback and review.

   • Decide whether to continue to the next loop, repeat the current loop, or terminate the project.

   • Deliverable: Updated requirements and feedback.

RAD (Rapid Application Development) Model

The Rapid Application Development (RAD) Model is a type of software development methodology that prioritizes rapid prototyping and quick delivery over long, detailed planning phases. It emphasizes user involvement and iterative development, allowing developers to quickly create a working prototype and refine it based on user feedback.

The RAD Model is particularly suited for projects where the requirements are well-defined, but the timeline for delivery is tight.

Phases of the RAD Model

The RAD Model consists of four main phases:

1. Requirement Planning:

   • Define high-level business objectives and scope of the project.

   • Identify core requirements and prioritize features.

   • Deliverable: Initial project roadmap.

2. User Design (Prototyping):

   • Collaborate with users to create prototypes representing key features.

   • Users interact with these prototypes to provide feedback.

   • Deliverable: Functional prototypes for testing and review.

3. Construction:

   • Rapidly develop the software using feedback from the prototyping phase.

   • Perform unit and integration testing.

   • Deliverable: Fully functional product increment.

4. Cutover:

   • Deploy the software to the production environment.

   • Train users, provide documentation, and handle final testing.

   • Deliverable: Deployed software and user-ready system.

V-Model (Validation and Verification Model)

The V-Model (Validation and Verification Model) is a software development methodology that extends the Waterfall Model by emphasizing testing at each phase of development. For every development stage on the "left side" of the V, there is a corresponding testing phase on the "right side" of the V. This approach ensures early detection of defects and aligns development activities with validation processes.

It is called the V-Model because the process diagram resembles the letter "V," with development activities descending on the left, meeting at coding, and ascending with validation/testing on the right.

Phases of the V-Model

Verification Phases (Left Side of the V):

1. Requirement Analysis:

   • Gather and document user needs and system requirements.

   • Corresponding Testing Phase: Acceptance Testing.

2. System Design:

   • Define the overall system architecture and design.

   • Corresponding Testing Phase: System Testing.

3. High-Level Design:

   • Break down the system into modules and specify their interactions.

   • Corresponding Testing Phase: Integration Testing.

4. Detailed Design:

   • Define the internal logic and structure of each module.

   • Corresponding Testing Phase: Unit Testing.

5. Implementation (Coding):

   • Write the code based on the detailed design specifications.

   • The base of the V marks the completion of development and the start of testing.

Comparison Table: AWS Parameter Store vs. AWS Secrets Manager

When to Use:

First 6 Months: Build Your Foundation

Start by understanding the core principles of DevOps, its culture, and the technical basics. These books are perfect for laying a strong foundation:

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, and George Spafford

  • Learn the fundamentals of DevOps through an engaging story about overcoming IT challenges.

• Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale by Jennifer Davis and Katherine Daniels

  • Discover the importance of culture, collaboration, and trust in successful DevOps teams.

• Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble and David Farley

  • Master CI/CD pipelines and the technical skills needed to automate testing and deployment.

6-12 Months: Dive Into Intermediate Concepts

Once you're comfortable with the basics, focus on cloud-native tools, infrastructure as code, and scaling DevOps practices:

• Cloud Native DevOps with Kubernetes: Building, Deploying, and Scaling Modern Applications in the Cloud by John Arundel and Justin Domingus

  • Get hands-on with Kubernetes and learn how to manage and scale containerized applications.

• Infrastructure as Code: Managing Servers in the Cloud by Kief Morris

  • Learn to treat infrastructure like code, using tools like Terraform and CloudFormation to manage cloud resources.

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations by Gene Kim, Patrick Debois, John Willis, and Jez Humble

  • This book provides detailed, actionable steps for applying DevOps principles in real-world environments.

1 Year and Beyond: Advanced Practices

Now it's time to tackle complex challenges like system reliability, scalability, and organizational transformation:

• Site Reliability Engineering: How Google Runs Production Systems by Niall Richard Murphy, Betsy Beyer, Chris Jones, and Jennifer Petoff

  • Dive into Google’s approach to reliability, scalability, and incident management.

• Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren, Jez Humble, and Gene Kim

  • Understand the key metrics that drive high-performing teams and learn how to scale DevOps practices effectively.

• Lean Enterprise: How High-Performance Organizations Innovate at Scale by Jez Humble, Joanne Molesky, and Barry O’Reilly

  • Learn to balance speed and stability while driving innovation and scaling practices in large organizations.

Specialized Focus: Enterprise-Level DevOps

For those working in hybrid IT environments or multi-speed enterprises, these books offer valuable strategies:

• The DevOps Adoption Playbook: A Guide to Adopting DevOps in a Multi-Speed IT Enterprise by Sanjeev Sharma

  • Navigate the challenges of adopting DevOps in enterprises with a mix of legacy and modern systems.

• The Unicorn Project: A Novel About Developers, Digital Disruption, and Thriving in the Age of Data by Gene Kim

  • Gain the developer’s perspective on thriving in a world of digital disruption and collaboration challenges.

Why This Roadmap Works

This timeline takes you step by step, from foundational knowledge to advanced expertise, providing practical insights at every stage of your DevOps career. By combining cultural understanding with technical skills, you'll be prepared to thrive in any DevOps environment.

Happy reading and building! 🚀

Here’s why it’s a must-use:

✅ Test Safely: Evaluate policies without impacting live environments.

✅ Debug Quickly: Pinpoint why a specific action is allowed or denied.

✅ Boost Security: Fine-tune permissions for the principle of least privilege.

As a Cloud/DevOps Engineer, this tool has been a game-changer in keeping my cloud infrastructure both secure and functional.

Tutorial:

https://lnkd.in/eeRbi5Ku

AWS Policy Simulator Tool:

https://lnkd.in/ePUvSZ3j

Here is what the AWS IAM Policy Simulator looks like:

𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬:

• Online statistics for server indicators such as CPU usage, memory usage, load average, and disk usage.

• Online ChatGPT Assistant

• One-click deployment and automatic renewal Let's Encrypt certificates.

• Online editing websites configurations with our self-designed NgxConfigEditor which is a user-friendly block editor for nginx configurations or Ace Code Editor which supports highlighting nginx configuration syntax.

• Online view Nginx logs

• Written in Go and Vue, distribution is a single executable binary.

• Automatically test configuration file and reload nginx after saving configuration.

• Web Terminal

• Dark Mode

• Responsive Web Design

Demo:

https://demo.nginxui.com

Username：admin

Password：admin

GitHub:

https://github.com/0xJacky/nginx-ui

A company recently faced this issue with their microservices on Amazon ECS behind an Application Load Balancer (ALB). Certain user requests were dragging down performance. 😟

Time to dig deeper!

The Solution: AWS X-Ray 🔍

Here's the fix:

1️⃣ Create a Docker image running the 𝐀𝐖𝐒 𝐗-𝐑𝐚𝐲 𝐝𝐚𝐞𝐦𝐨𝐧.

2️⃣ Run it alongside your microservices in ECS.

3️⃣ Use the X-Ray console to analyze request behavior and pinpoint performance bottlenecks. 🎯

X-Ray’s distributed tracing helps you uncover hidden issues and optimize your app across services. ✨

👉 Learn more about running the X-Ray daemon on Amazon ECS:

Running the X-Ray daemon on Amazon ECS - AWS X-Ray

Understanding the Unpredictability of Real-World Traffic

In a real-world scenario, it is often impossible to predict exactly how many visitors your site will have or how many reads and writes they will generate. Traffic could spike unexpectedly (e.g., due to a sale or a social media promotion), and trying to predict that traffic precisely can be like guessing the weather weeks in advance.

1. Estimating Capacity with Assumptions

To help you plan your read and write capacity, you can make some initial assumptions based on your website's expected usage. Here’s how you can think about it:

Example Scenario: Online Product Store

Let's say you are building an e-commerce site and using DynamoDB to store information about products and customer orders. You need to think about both reads and writes to estimate the capacity:

• Reads: Customers browsing products are generating read requests.

• Writes: When a customer places an order, or a new product is added, that is a write request.

Estimating Reads:

• You estimate that there will be 1,000 unique visitors to your website per day.

• On average, each visitor views 10 product pages. This means you would have:1,000 visitors * 10 reads per visitor = 10,000 reads per day.

Now let’s estimate reads per second:

• Divide 10,000 reads per day by the number of seconds in a day (86400 seconds).

• 10,000 / 86,400 ≈ 0.12 reads per second. You can round this up to 1 read per second to be safe.

Estimating Writes:

• Assume that 10% of visitors place an order.

• This means you would have:1,000 visitors * 10% = 100 orders per day.

Now let’s estimate writes per second:

• Divide 100 writes per day by the number of seconds in a day (86400 seconds).

• 100 / 86,400 ≈ 0.001 writes per second. Again, round up to 1 write per second for safety.

Based on this, you could initially configure your DynamoDB table for:

• 1 Read Capacity Unit (RCU) per second.

• 1 Write Capacity Unit (WCU) per second.

2. Capacity Modes in DynamoDB

Instead of having to estimate exactly, DynamoDB offers two capacity modes that are helpful in dealing with unpredictable workloads:

Provisioned Capacity Mode:

• This is where you set a fixed number of RCUs and WCUs based on your estimates, as described above.

• Pros: If you have relatively predictable traffic, it allows you to control costs by paying only for what you need.

• Cons: If your traffic spikes suddenly and goes beyond your provisioned capacity, you may experience throttling (i.e., some requests may fail).

On-Demand Capacity Mode:

• On-Demand Mode means you don't need to specify the number of RCUs or WCUs upfront. DynamoDB will automatically handle scaling based on incoming traffic.

• Pros: This is ideal for unpredictable workloads where you might get spikes in traffic.

• Cons: It's more expensive per operation compared to provisioned capacity, but it prevents throttling during spikes.

Real-World Strategies for Handling Capacity

Since predicting exact traffic is challenging, many companies use a combination of the following:

1. Start with On-Demand Mode:

• If your workload is unpredictable, it’s better to start with on-demand capacity mode. This ensures that DynamoDB can scale automatically to handle whatever traffic comes to your website, and you won’t have to deal with capacity planning initially.

• This is useful during initial launches, product promotions, or campaigns when it's unclear how much traffic to expect.

2. Monitor Usage Patterns:

• Use AWS CloudWatch to monitor the number of reads and writes on your DynamoDB table.

• Once you have a better understanding of your traffic patterns (e.g., after a few weeks or months), you could potentially switch to provisioned capacity if your traffic becomes more predictable, which helps lower your costs.

3. DynamoDB Autoscaling (Provisioned Capacity):

• If you choose provisioned capacity mode, you can also enable autoscaling. This allows DynamoDB to automatically adjust the RCUs and WCUs up or down based on the traffic.

• You set a minimum and maximum threshold so that capacity scales within a defined range, avoiding both under-provisioning (which causes throttling) and over-provisioning (which increases cost).

Summary

Predicting Traffic: It’s nearly impossible to predict exact traffic in the real world, especially at launch. You can start by estimating based on assumptions, but this is just an approximation.

Capacity Planning Modes:

• Use on-demand capacity if you have unpredictable traffic or expect spikes.

• Use provisioned capacity if your workload is predictable, and consider autoscaling to adjust as needed.

Real-World Approach: Many people start with on-demand capacity to avoid worrying about underestimating traffic. Once they understand their traffic patterns and the average number of reads and writes, they may switch to provisioned capacity with autoscaling to optimize for cost.

In simple terms, managing read and write capacity units in DynamoDB is all about making sure your table can handle your expected traffic without throttling, and the key is to balance cost with availability by choosing the right capacity mode based on your application's needs.