The developers are actively build and test their projects. There are dozens of projects that can be built in a day. This sometimes causes the disk of the CI/CD system (e.g., Jenkins, self-hosted runner) to be full, so we always delete unused images and build cache after each build. When a project needs to be rebuilt, it has to pull the base images from Docker Hub. But there is a rate limit from dockerhub, 100 pulls/6h unauthenticated, 200 authenticated per IPv4 address. If we use more than this, the build pipeline will fail.

The Solution

Because I use AWS, I can utilize AWS ECR pull-through cache to pull images from dockerhub, store them in ECR, then I can pull unlimited images from ECR. No dependency on Docker Hub uptime/rate limits. If Hub goes down, you can still pull (assuming the image you need is already cached on ECR). However, ECR storage cost for cached images. Mitigate it with lifecycle policies to evict old/unused tags.

Create a Docker Hub personal access token

This will be used by ECR to login to the Dockerhub. Token is used instead of your Docker hub password.

1. Log into Docker Hub

2. Navigate to Account Settings > Settings > Personal access tokens

3. Click the "Generate new token" button

4. Fill the token description, set the expiration date, and the permissions. Usually, I only give read-only permission.

5. Write down the generated PAT for the next step in AWS configuration.

Setting up the ECR pull-through rule

1. First, you need to store your Docker Hub credentials in AWS Secrets Manager with a specific naming convention. Use this AWS CLI command:

   aws secretsmanager create-secret \
       --name "ecr-pullthroughcache/docker-hub" \
       --description "Docker Hub credentials for ECR pull through cache" \
       --secret-string '{
           "username": "<your-docker-username>",
           "accessToken": "<your-docker-access-token>"
       }' \
       --region <region>
   

   Important: The secret name must use the ecr-pullthroughcache/ prefix and be in the same account and region as your pull-through cache rule.

2. Proceed with creating the pull-through rule. You can use this AWS CLI command:

   aws ecr create-pull-through-cache-rule \
       --ecr-repository-prefix docker-hub \
       --upstream-registry-url registry-1.docker.io \
       --credential-arn <secretsmanager-arn-you-got-from-the-previous-command> \
       --region <region>
   

3. Or, if you prefer to use the console, navigate to ECR > Private Registry > Pull through cache > Add Rule.

   

4. Set the Upstream registry to Docker Hub.

5. In the Authentication section, select the Use an existing AWS secret and select the secret created in step 1. Note that you can also skip step 1 and create the secrets from here if you use the console.

6. In the Step 3: Specify namespaces, configure the prefix configuration for both the cache and upstream namespace. I usually select use a specific prefix for the namespace, and no prefix in the Upstream namespace.

7. On Step 4: Review and create, review your configuration and choose Create

Pull the image through the cache

# Login to the ECR private registry
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <your-acc-id>.dkr.ecr.<region>.amazonaws.com

# Pull an image from docker hub:
docker pull aws_account_id.dkr.ecr.region.amazonaws.com/docker-hub/library/image_name:tag

Notice the library in the repository URL is specific to Docker Hub's official images and reflects how Docker Hub organizes its repositories internally.

# Original Docker Hub command:
docker pull nginx:latest
docker pull alpine:3.23.4

# ECR pull-through cache equivalent:
docker pull aws_account_id.dkr.ecr.region.amazonaws.com/docker-hub/library/nginx:latest

When you pull these images, ECR creates repositories with these exact names:

• docker-hub/library/nginx

• docker-hub/library/alpine

Setting up the Repository Policy

I personally use a repository policy template to automatically apply a policy when ECR create Dockerhub pull-through repository. Such as, to only keep the last 6 images.

1. Navigate to ECR > Private Registry > Features & Settings > Repository creation templates > Configure > Create Template.

   

2. On Step 1: Define template > Template details > Applied for select the Pull through cache. Because I use a prefix in the previous step, I select the A specific prefix and set the prefix there.

3. On Step 2: Add repository creation configuration > Repository lifecycle policy, you can select predefined templates in the Lifecycle policy examples.

4. On Step 4: Review and create, review your configuration and choose Create

Now, if you pull a Dockerhub image through ECR, a lifecycle policy will be automatically applied.

Today, I need to export a MySQL database and then import it to a new server. The database is not too big, just a few GBs. But it was taking so long, and I wonder when this will be finished? By default, when you run mysqldump command to export the database and then mysql to import it, it doesn’t show any progress. I'm not sure whether the process is stuck or running.

root@server:~# mysqldump some_database > backup.sql

# Nothing to see here.
# The terminal shows nothing until the process is finished.

The solution

Then I discovered Pipe Viewer (PV). According to the website, Pipe Viewer is:

Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline and modifying its flow. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. Data flow rate, error handling strategy, buffer size, and cache interaction can all be adjusted.

Let’s see how it works:

# Install the pv package
root@47be583c9aaf:/# apt update && apt install -y pv

# Let's export a database.
# Using mysqldump, there’s no "total expected size".
# While PV can show progress percentages only when it knows the total size.
# I know the DB size. So I tell PV using the "-s" option.
root@localhost:~# mysqldump some_database | pv  --size 3G > some_database.sql
36MiB 0:00:09 [65.8MiB/s] [========>                           ] 17% ETA 0:00:42

# Now let's import the database
# Now PV know the size of the DB dump, so I don't need the "-s" option.
root@47be583c9aaf:~# pv some_database.sql | mysql -p some_database
5.09GiB 0:06:47 [12.8MiB/s] [==================================>] 100%

Now I don’t have to worry, knowing when it will be finished and the process speed.

PV is not exclusive to MySQL-related activity. But it can also be used for other use cases. Such as monitoring the progress of file compression and decompression, copying a file, and many more.

Current setup

I have a Redis OSS instance on AWS ElastiCache configured like this:

• Cluster mode: Disabled (single shard)

• Nodes: 1

• Auto-failover: Disabled

• Multi-AZ: Disabled

In short, it’s basically a standalone Redis instance that is enough for my application. For some reason (my laziness + “it’s working so why change it” + lack of reading documentation), my app was connecting directly to the Node Endpoint instead of the Primary Endpoint.

A small change started the issue

I changed some of the settings, including turning on the Encryption in Transit, set it to Preferred, and checking if I’m still able to connect to the Redis (somehow I'm still able to connect to the Redis using the old DNS). Seemed simple enough, right? No, it’s not.

The realization

What I didn’t realize is that changing this setting forces AWS to change the Node Endpoint DNS. The AWS documentation here clearly said:

Unlike the primary endpoint, node endpoints resolve to specific endpoints. If you make a change in your cluster, such as adding or deleting a replica, you must update the node endpoints in your application. There is a difference depending upon whether or not In-Transit encryption is enabled.

In-transit encryption not enabled

clusterName.xxxxxx.nodeId.regionAndAz.cache.amazonaws.com:port

example: redis-01.7abc2d.0001.usw2.cache.amazonaws.com:6379

In-transit encryption enabled

master.clusterName.xxxxxx.regionAndAz.cache.amazonaws.com:port

example: master.ncit.ameaqx.use1.cache.amazonaws.com:6379

The result was predictable. My app was still trying to connect to the old node hostname, but Redis was gone from that address. And the downtime occurred.

Which endpoints to use with Valkey or Redis OSS?

The AWS documentation says:

For a standalone node, use the node's endpoint for both read and write operations.

AFAIK, your Redis instance is considered Standalone if there is no replica, cluster mode is disabled, and encryption in transit is disabled (correct me if I’m wrong). As you can see here, there is only a reader endpoint. So, you'd better connect directly to the node endpoint.

For Valkey or Valkey or Redis OSS (cluster mode disabled) clusters, use the Primary Endpoint for all write operations. Use the Reader Endpoint to evenly split incoming connections to the endpoint between all read replicas. Use the individual Node Endpoints for read operations (In the API/CLI these are referred to as Read Endpoints).

If you enable Encryption in transit, AWS will add a Primary Endpoint. They suggest you use Primary Endpoint for write operations, a Reader Endpoint or you can connect directly to the individual Node Endpoints for read operations. But I, myself, prefer connecting only to the Primary Endpoint for both write and read operations.

For Valkey or Redis OSS (cluster mode enabled) clusters, use the cluster's Configuration Endpoint for all operations that support cluster mode enabled commands. You must use a client that supports either Valkey Cluster, or Redis OSS Cluster on Redis OSS 3.2 and above. You can still read from individual node endpoints (In the API/CLI these are referred to as Read Endpoints).

I haven’t tried this yet, but I believe you can figure it out yourself.

Lesson learned

• Make your client resilient: retry, reconnect, and respect DNS TTLs.

• Test thoroughly and pay attention to details before you make changes to the production for real.

• Plan disruptive changes (like enabling TLS) with a maintenance window.

• Always read the official documentation.

• Connect to the appropriate endpoints depending on your use case. I prefer using the Primary Endpoints for both write and read operations. This may result in read operations not being distributed evenly across all replicas. But I don't have any replicas, and cluster mode is disabled.

What is KCNA certification?

The Kubernetes and Cloud Native Associate (KCNA) certification is an entry-level credential designed to validate foundational knowledge of Kubernetes, containers, and the broader cloud native ecosystem. It’s ideal for beginners, offering a solid introduction to core concepts like pods, deployments, container runtimes, and CNCF projects.

People generally take this certification after completing the CKAD or CKA certifications because they already understand the basics, making it easier to pass the KCNA. Or, for those new to Kubernetes, this certification can be a starting point for learning Kubernetes and Cloud Native.

The exam consists of 60 multiple-choice questions that must be completed within 90 minutes. You will pass if you achieve a score above 75. One retake is included, so don’t worry if you fail once.

My background

Maybe you need to know about me first because the tips I share may not be suitable for you. I've worked in IT for the past 7 years. I've worked as a Linux Sysadmin, Network Engineer, SRE, and DevOps. So, I'm quite familiar with the cloud and DevOps worlds. I interact with cloud services like AWS and GCP every day. I use Docker every day and often use Kubernetes, but I don't mess with it much. That way, it didn't take me too long to prepare for this exam.

What to learn?

You can start by studying the KCNA curriculum, which can be read at this link. I prioritized these three chapters as essential to study because they account for the largest percentage of the learning:

1. Kubernetes Fundamentals - 46% (The most important and the longest I studied).

2. Container Orchestration - 22% (Kubernetes complementary components).

3. Cloud Native Architecture - 16% (Mostly related to standards in Cloud Native and getting to know the CNCF organization).

Where to learn?

I generally use these study materials:

1. KCNA certification + Hands-on Lab + Practice Exam, Udemy course by James Spurin. I found the explanations very easy to understand, clear, and concise. There is a quiz after each chapter, so what you have learned doesn’t evaporate quickly. There's a hands-on section to get a better feel for using Kubernetes. Two practice exams and many more quizzes, which I found to be very similar to the actual exam environment. See it by yourself.

2. Becoming KCNA Certified book by Dmitry Galkin. If you prefer reading books to watching videos.

3. KCNA-training repository by edithturn.

4. Of course, the Kubernetes official documentation.

5. The CNCF website and CNCF Landscape.

Scheduling the exam

Scheduling an exam is quite straightforward. Be sure to read the System Requirements and Testing Environment Requirements. Failure to meet either of these requirements could prevent you from starting the exam or even result in disqualification.

The links above or the exam rules may become outdated in the future. Make sure to follow the instructions on the registration page. Here are some requirements I pay special attention to:

1. Prepare a quiet place where no other people are coming in and out.

2. The table and its surroundings must be clean of paper, stationery, and other electronic devices such as smartphones.

3. There must be an active webcam and mic during the exam process.

4. I recommend using Windows 11 for an easier setup. Use a laptop and have a data plan ready in case of a power outage.

5. The PSI browser download link will be available after scheduling the exam.

6. Take the exam simulation at least once to familiarize yourself with the exam conditions.

7. Be sure to check the exam rescheduling policy.

Taking the exam

I took the exam at midnight (11.30 PM UTC+7). The exam link will be available 30 minutes before the scheduled time. Be sure to start the exam as soon as possible because there will be a data verification process that may take a long time. Make sure your ID card (such as Indonesian KTP) is within reach. I joined the exam 15 minutes before it started yesterday (11.17 PM UTC+7). It turned out I only managed to complete it 30 minutes after the scheduled time (00.01 AM UTC +7). This is my setup during the exam:

Pay special attention to these rules:

1. I’m allowed to bring a bottle of water to drink. The bottle must be clear of labels or writings.

2. You communicate with the proctor or support via chat in English.

3. I can ask for a short break, but you must stay in your seat.

4. I'm not allowed to wear headphones or other accessories like hats.

5. Avoid covering your mouth, mumbling, or reading the question aloud when working on questions.

6. Avoid looking away from the screen.

General tips

These are tips that work for me. I believe these tips also work for other certification exams:

1. Ask for prayers from your mother or wife (if married). Treat them well.

2. Study 50% - 60% of the material that will help you understand the whole material.

3. Do hands-on exercises to make it easier to understand Kubernetes. Create a small project like deploying Wordpress on Kubernetes.

4. The closer to the exam date, the more I focus on practicing questions.

5. Don't spend too much time on a single question. Limit it to a maximum of 5 minutes. If you're unsure, you can mark the question and review it later.

6. Use a mnemonic (“jembatan keledai” in Indonesia) to help remember information.

7. Don't just read and listen to the material. You can explain (verbally or in writing) what you've learned so you don't forget it quickly.

8. Use multiple resources. Avoid believing solely in AI. Always verify information.

9. Don't fear mistakes. Learn from them. There is one question that often bamboozled me. I chose the right answer multiple times when practicing the questions, and I still second-guessed myself during the exam and chose the wrong answer. Yes, I’m still mad about it. But making mistakes is the most effective way to actually learn because it annoys the hell out of you.

10. The score you get won’t matter, and won't appear on your certificate. What matters is that you passed the exam. I'm saying this so you don't focus too much on getting a perfect score.

Final Thoughts

KCNA is one of the great ways to get started with the cloud, the Cloud Native, and the Kubernetes world, especially if you want to work in the DevOps field. I’m sure it’s not the hardest exam, but it’s a step toward becoming a more confident and capable engineer who understands Kubernetes fundamentals. This exam isn’t about becoming a Kubernetes guru; it’s about proving you understand the ecosystem, the tools, the culture, and the direction the cloud-native movement is heading.

And don’t think too much about a perfect score. ChatGPT told me this after I made a trivial mistake:

Don't chase a perfect score too much. A perfect score is sweet, but the real flex is knowing your stuff so well that one tiny misstep doesn’t shake your confidence.

Remembering IP in a massive, dynamic environment is not easy. You may have an instance with IP 10.0.0.1 today, but there is no guarantee that the same server will be there tomorrow. If you are still SSH-ing to your server using a traditional method like ssh user@ip_addressyou’ll have a hard time remembering the IP address of that server. Wouldn't it be easier to just run server_a or server_b to get into that particular server? You just have to know the name of the server, which is easier to remember. Also, the command is dynamically updated when a new server is created or deleted.

tl;dr

• Allow the bastion host to read EC2 metadata

• Create a Python script to get all the instance names and private IPv4 addresses

• Add the command alias to your .bashrc or .zshrc file.

• Add cron to run the script automatically and update your alias.

Allow Bastion host to get EC2 instances

1. The script will get all the EC2 data using the describe-instances command. Create an AWS IAM policy to allow read access to the instances:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "ec2:DescribeInstances",
          "Resource": "*"
        }
      ]
    }
   

2. Create or modify your existing Bastion IAM role to attach this IAM policy.

3. Verify the access by running aws ec2 describe-instances inside the Bastion host.

Create a script to generate the command alias

The idea is to generate this output that can be fed to the .bashrc:

alias server_a="ssh ubuntu@private_ipv4_address"
...

Here’s the Python code to do that:

Generate the alias and write it to ~/.awsvmaliases:

python3 aliasgen.py > ~/.awsvmaliases

Automatically load the aliases

To automatically load those aliases when you open the terminal, open your .bashrc or .zshrc file and append this line:

...
source ~/.awsvmaliases

Automatically update the list

In my case, it is sufficient to update the script once an hour. So I just use cron to update the aliases:

# Inside the crontab
@hourly /usr/bin/python3 path/to/aliasgen.py > ~/.awsvmaliases

If you need to manually update the list without waiting for cron to run, you can run this command:

python3 path/to/aliasgen.py > ~/.awsvmaliases
# If you use bash shell, tell bash to reload the .bashrc file and read the changes
source ~/.bashrc

• "Apply Immediately" applies everything in the pending modifications queue, not just your change.

• Always check pending modifications and maintenance tabs first.

• Don't accidentally trigger a db-upgrade unless you're ready for downtime.

What just happened?

Today I was preparing for a planned maintenance, just an hour before the scheduled time. We planned to upgrade one of our RDS instance to the latest minor version. It came the time when I needed to change the Parameter Group. It looked like a simple change and shouldn’t cause any downtime except when you reboot the instance to apply the changes. I have done this several times before and I can confirm it stays that way.

I checked my modifications again just to make sure I didn’t misclick anything. Auto minor upgrade was disabled, and the maintenance window wasn’t scheduled today. I chose “Apply Immediately”, assuming it would apply only to my intended modification. Instead, I got unexpected downtime. AWS started showing “Upgrading” status instead of “Available”, and the database went offline for a few minutes. Tried to figure out what just happened and keep calm while investigating how it happened. Turns out it was my modification that triggered the pending mandatory engine upgrade to run immediately.

So, how did that happen?

When the instance was still in the Upgrading state, I realized there was a required db-upgrade in the Maintenance & Backup tab. It wasn’t scheduled today, so why did AWS apply it now? Turns out when you choose to apply the modification immediately, AWS RDS will think “Oh, the user is changing something. We think it’s a good time to apply any other pending modifications as well”, instead of just what I want.

According to the official AWS documentation:

If you don't choose to apply changes immediately, RDS puts the changes into the pending modifications queue. During the next maintenance window, RDS applies any pending changes in the queue. If you choose to apply changes immediately, your new changes and any changes in the pending modifications queue are applied.

So if you have any required maintenance actions queued up, like a minor version upgrade, they will be applied instantly along with your changes, even if you didn’t intend to touch the engine version. And yes, a db-upgrade restarts your instance, which means downtime.

Check before you fall into the trap

tl;dr: Don’t Apply Immediately Blindly.

Before you make any changes that need to be applied immediately, always check for pending queue for that RDS instance! There may be a pending modification or pending maintenance action. Here’s how you do it:

From Console

1. Go to RDS console and find your instance.

2. Look for Maintenance column.

3. Click the “Maintenance & Backups” tab.

4. Look for any changes listed in the “Pending maintenance” and “Pending modifications”.

From AWS CLI

1. Look for pending modifications:

    aws rds describe-db-instances \
      --db-instance-identifier your-db-name \
      --query "DBInstances[*].PendingModifiedValues"
   

2. Look for your instances in this list of pending maintenance actions:

    aws rds describe-pending-maintenance-actions --region <your region>
   

Undo or cancel changes

AFAIK, I haven’t found a way to cancel, defer, or make immediate changes without triggering the required update or changes in the queue. But you can cancel some of the non-required pending modifications by following this answer from Stack Overflow.

Final thoughts

AWS managed services, such as RDS, take a lot of the burden off an engineer’s shoulders. But it requires you to understand how it works. Otherwise, we might run into a problem similar to what I experienced. In my opinion, AWS should give the option to choose which modification will be applied instead of applying all pending modifications at once. Or just a list of what modifications (including in the queue) will be made before I click that final “Modify”.

Luckily, in my case, it was nighttime and only a few people were using the database, so the incident didn’t cause significant harm.

GCP Cloud Services

Compute

• Compute Engine: Ini layanan VM-nya GCP yang masuk kategori IaaS. Cocok digunakan jika kita ingin punya kontrol penuh atas resource ini. Misalnya seberapa besar resourcenya, pakai CPU apa, pilih OS sesuai kebutuhan, mau diinstall apa, kapan harus dibackup, kapan harus diupgrade, dan aktivitas lainnya. Downside-nya adalah karena layanannya masih cukup mentah, kita harus paham dengan hal seputar administrasi server.

  Kelebihan lainnya adalah adanya opsi Spot VM yang memberikan diskon 60-91% dari harga VM biasa. Konsekuensinya adalah VM tersebut bisa kapan saja distop atau didelete oleh GCP. Cocok untuk fault-tolerant workload dan batch processing yang mungkin cuma butuh waktu sebentar untuk bekerja.

• Kubernetes Engine: Dipakai untuk ngejalanin banyak container diatas Kubernetes cluster yang berjalan diatas banyak VM. GCP akan nge-handle resource provisioning, node autoscaling, health check, dan replace unhealthy server secara otomatis. Intinya, nge-maintain Kubernetes cluster bakal lebih mudah kalau pakai GKE daripada langsung manage lewat Compute Engine. Sebenarnya GKE juga menggunakan Compute Engine di belakang layar sebagai worker nodenya.

• App Engine: Ini layanan PaaS dari GCP. App Engine memudahkan developer untuk menjalankan aplikasinya tanpa ribet urusan server. Sebenarnya tetep pakai Compute Engine di balik layar, tapi urusan administrasi server, konfigurasi, dan network hardeningnya sudah dihandle GCP. Jadi pricingnya ngga jauh-jauh dari Compute Engine. AFAIK konsep kasarnya kayak Heroku atau Vercel yang sama-sama PaaS tapi versi lebih powerfull. Atau mirip kayak AWS ElasticBeanstalk. Ada 2 tipe App Engine:

  • Standard Environment: App yang udah kita buat akan dijalankan di dalam language-specific container yang disupport GCP seperti Java, Go, Node.js, PHP, Python, dan Ruby. Jadi kalo kita butuh selain itu, bisa pilih Flexible environment yang mendukung custom runtime. Standard Environment cocok dipakai jika app kita ngga butuh OS packages atau software tambahan untuk menjalankan appnya.

  • Flexible Environment: Beda dengan standard environment yang punya language constraint. Flexible environment bisa menjalankan Python, Java, Node.js, Go, Ruby, PHP, .NET, dan any software that can service HTTP requests (custom runtime). Bisa specify custom Dockerfile juga.

• Cloud Function: Layanan serverless untuk menjalankan sebuah function yang melakukan satu. Cocok digunakan untuk event-driven processing, lightweight computing, atau execute short-running code. In my opinion, ini mirip kayak AWS Lambda. Salah satu use-case Cloud Function adalah ketika sebuah object diupload ke Cloud Storage, maka sebuah Cloud Function akan dipanggil untuk melakukan sesuatu. Misalnya membuat thumbnail, memproses object tadi, dan lain-lain. Function mensupport Node.js, Python, Go, Java, Ruby, PHP, dan .Net.

Storage

Block Storage itu storage mentah semacam hard drive di komputer yang masih kosong dan belum ada filesystemnya. Masih perlu diolah sebentar agar bisa digunakan. Karena masih mentah, kita bisa bebas pilih filesystem (misal ext4, xfs, dst.), diatur kayak gimana (misal partisi, size, dst.), dan dipakai untuk apa (misal diinstall OS, backup storage, dst.).

Object Storage itu layanan penyimpanan file yang udah siap pakai. Paling sering dipakai untuk menyimpan file yang diupload dari suatu aplikasi baik mobile, desktop, maupun web. Di sini file disebut dengan Object. Setiap Object akan dikumpulkan dalam suatu Bucket dan bisa diakses menggunakan suatu URL yang unik. Layanan ini Serverless, jadi gak perlu manage server dan software storagenya. Object Storage bisa diakses dari banyak VM.

File Storage itu layanan network shared file systems (sorry, agak bingung jelasin gampangnya gimana). Jadi pakai protokol NFS untuk nge-akses filenya. File Storage juga udah siap pakai. Tinggal memastikan sistem kita sudah support NFS. Tipe storage ini bisa diakses oleh banyak VM dalam waktu bersamaan.

• Cloud Storage (GCS): Layanan Object Storage di GCP seperti layanan S3 di AWS. Hak akses ke file dan bucket bisa diatur lewat IAM. Cloud Storage cocok digunakan untuk menyimpan semua jenis file seperti gambar, video, dokumen, dan lain-lain. Contoh use-casenya adalah sebuah perusahaan dapat menggunakannya untuk menyimpan foto dan video yang sudah diupload oleh pengguna lewat aplikasi yang dibuat perusahaan itu.

• Persistent Disk (PD): Layanan Block Storage di GCP. Ada pilihan SSD dan HDD. Yang menarik adalah PD bisa dipasang ke banyak VM sekaligus. Maksimal kapasitas PD adalah 64TB. Ukurannya bisa di-resize tanpa downtime seperti di artikel ini.

• Cloud Storage for Firebase: Object storage khusus untuk aplikasi yang terintegrasi dengan Firebase. Cocok digunakan untuk upload dan download di aplikasi mobile yang jaringannya sering ngadat karena punya mekanisme recovery khusus. Sebenarnya pakai GCS di balik layar. Dia juga udah terintegrasi dengan Firebase Auth.

• Cloud Filestore: Layanan File storage yang bisa diakses (read-write) di banyak VM sekaligus pakai protocol NFS. Maksimal kapasitas Filestore adalah 100 TB, throughputnya 25 GB/s, dan IOPS-nya 920K.

Database

• Cloud SQL: Layanan RDBMS dari GCP. Support MySQL, PostgreSQL, dan SQL server. Banyak fitur untuk mengelola RDBMS seperti backup, patching, network configuration, monitoring, replication, failover, dan lain-lain. Cocok digunakan untuk mempermudah proses pengelolaan RDBMS dibanding menjalankannya di VM biasa.

• Cloud Spanner: Layanan RDBMS tapi globally scalable dan support standar SQL ANSI 2011.

• Cloud Bigtable: Layanan NoSQL jenis wide-column database yang didesain untuk mengelola database skala besar. Di soal latihan GCP ACE, biasanya dia dipakai untuk menerima dan menyimpan data dari ratusan sensor IoT dengan aktivitas write yang kecil-kecil tapi banyak banget. Atau bisa juga untuk real-time analytics.

• Cloud Datastore: Layanan NoSQL jenis document database. Dia otomatis ngehandle sharding dan replikasi serta support transaction, index, dan query seperti SQL. Beberapa use-casenya adalah katalog produk yang menampilkan detail produk secara real-time dan user profile di suatu aplikasi yang personally customized berdasarkan aktivitas user dan preferensinya. Datastore beda dengan Bigtable dimana Bigtable lebih cocok digunakan untuk aktivitas read/write yang banyak dan bersamaan dengan latensi kecil.

• Cloud Firestore: Next generation-nya Datastore dan Google lebih menyarankan menggunakan ini (beda dengan Filestore ya!). Firestore itu lebih modern dan scalable dibanding Datastore serta kompatibel dengan MongoDB.

• Cloud Memor**y**store: Layanan serverless dan full-managed in-memory database seperti Redis, Valkey, dan Memcache. Datanya disimpen di memory (kayak RAM) jadi bisa diakses jauh lebih cepat dibanding dari hard drive. Cocok untuk nge-cache data yang sering diakses biar nggak membebani server databasenya. Full-managed itu artinya masalah scalability, high availability, patching, dan failover udah dihandle GCP sendiri.

Networking

• Virtual Private Cloud (VPC): Layanan networking yang memungkinkan kita mendesain network data center kita sendiri seperti alokasi IP, subnetting, firewall, dan lain-lain.

• Cloud Load Balancing: Mendistribusikan workload di dalam suatu region maupun globally antar region. Dia juga menyediakan fitur autoscaling dan auto-repair ketika ada server yang bermasalah. LB mendukung protokol HTTP, HTTPS, TCP, dan UDP.

• Cloud Armor: Layanan network security berupa DDoS protection dan WAF.

• Cloud CDN: Layanan Content Delivery Network (CDN) yang mempercepat akses suatu konten dari mana saja.

• Cloud Interconnect: Layanan untuk menyambungkan jaringan kita ke data center GCP secara langsung. Jadi dipakai kalau kita punya data center sendiri dan ingin nyambung secara fisik (kabel nyambung) ke data center GCP. Ini sebutannya adalah Direct Interconnect. Nah kalau gak bisa nyambung secara fisik, maka pilih Partner Interconnect dimana akan ada ISP perantara yang menghubungkan kita dengan GCP. Interconnect cocok digunakan kalau kita butuh koneksi yang low latency & high bandwidth ke resource yang ada di GCP.

• Cloud DNS: Ya layanan DNS sih, kayak Route53 di AWS. Memungkinkan kita untuk memanage record DNS dari domain yang kita kelola.

This Dockerfile Works... But Should You Use It?

# Mistake #1: Using a single stage (no build separation)
# This makes the image larger than necessary
# and enlarge the attack surface, as the final image includes the build tools
# and source code, which are not needed for the final image
FROM golang:1.21-alpine
WORKDIR /app

# Mistake #2: Using ENV instead of ARG and Hardcoded sensitive information
# This is a security risk, as ENV variables are stored in the image
# and can be accessed by anyone with access to the image
# Fortunately, Docker can detect hardcoded sensitive information
# and will warn you about it.
ENV DBPASSWORD=acompletelyinsecurepassword
ENV DBNAME=postgres
ENV DBUSER=postgres

# Mistake #3: Not cleaning up unnecessary files
# Copies everything, including .git, .env, and dev files
COPY . .

# There is a good dicsussion about CGO_ENABLED=0
# https://www.reddit.com/r/golang/comments/pi97sp/what_is_the_consequence_of_using_cgo_enabled0/
RUN go mod tidy && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o server .

# Mistake #4: Running the server as root or not specifying a user
# This is a security risk, as the server has full access to the system
# and can be used to escalate privileges
# In this example the default user for golang:1.21-alpine is root, which is a bad practice and should be avoided
# You can check by running the following command:
# docker run -it --rm golang:1.21-alpine whoami
USER root

# Mistake #5: Exposing unnecessary ports
# Exposes SSH and an additional port, which are not needed for the server
# and can be used to attack the system
EXPOSE 22 8080
CMD ["./server"]

Let’s Fix This Dockerfile

# Correction #1: Use multi-stage builds to reduce the image size
# and reduce the attack surface
FROM golang:1.21-alpine AS build
WORKDIR /app

# Correction #2: Use .dockerignore to exclude unnecessary files
# The copy command here is okay to do because when using multi-stage builds
# the final image will only contain the binary file and not the source code
COPY . .
RUN go mod tidy && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o server .

# Correction #3: Use smaller base image as the final image
# You can use Alpine, Scratch, Ubuntu, or any other small base image depending on your needs
# You can also use Distroless images for more security
# Using Scratch or Distroless images will not have a shell, package manager, or any other programs in typical Linux distributions.
# So if a hacker gains access to the container, they won't be able to do much
# However, those images are not recommended for development because they are harder to debug
# since you can't exec into the container and run commands.
# They also requires more technical knowledge to use.
# So to balance between security and ease of use, I'll use Alpine here
# https://medium.com/google-cloud/alpine-distroless-or-scratch-caac35250e0b
FROM alpine:3.21.3 AS final

# Correction #4: Use ARG instead of ENV to pass build-time variables
# This is because ARG is only available during the build stage
# and the values are not stored in the final image
ARG DBPASSWORD
ARG DBNAME
ARG DBUSER

# Correction #5: Use non-root user
# However, you need to make sure that the user has the necessary permissions to run the application
USER nobody:nogroup

WORKDIR /app
# The COPY command here is to copy the binary file from the build stage
# to the final image
COPY --from=build --chown=nobody:nogroup /app/server .

# Correction #6: Only expose the necessary port
EXPOSE 8080

CMD ["/app/server"]

By making the image secure, usually you ended up with a smaller image. From what I’ve tried, here is the result:

Increase the block storage

In this case my Ubuntu VM is running on AWS with storage of 8GB and need to increase it to 12GB. To increase the volume size:

1. Go to AWS EC2 Dashboard.

2. Click Volumes and find your volume that you want to increase.

3. Right click the volume name and choose Modify Volume.

4. Increase the volume size in the Size (GB) field.

Resize the partition

SSH into the server and run the following commands.

# Let's inspect the block devices first
# Here I have nvme0n1 disk.
# The disk size was 8GB and increased to 12GB.
# Then I want the nvme0n1p1 (first partition) to use the remaining free space (4GB)
ubuntu@i-00cfd0679ab3871a3:~$ lsblk 
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
...
nvme0n1      259:0    0   12G  0 disk 
├─nvme0n1p1  259:1    0    7G  0 part /
├─nvme0n1p15 259:2    0   99M  0 part /boot/efi
└─nvme0n1p16 259:3    0  923M  0 part /boot

# Let's increase it using growpart
# https://access.redhat.com/solutions/5540131
ubuntu@i-00cfd0679ab3871a3:~$ sudo growpart /dev/nvme0n1 1
CHANGED: partition=1 start=2099200 old: size=14677983 end=16777182 new: size=23066591 end=25165790

# Check the partition again
# Make sure the nvme0n1p1 size is increased
ubuntu@i-00cfd0679ab3871a3:~$ lsblk 
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
...
nvme0n1      259:0    0   12G  0 disk 
├─nvme0n1p1  259:1    0   11G  0 part /
├─nvme0n1p15 259:2    0   99M  0 part /boot/efi
└─nvme0n1p16 259:3    0  923M  0 part /boot

growpart is part of utility to extend a storage in cloud environment. Usually it’s already installed in the OS image as part of cloud-guest-utils package.

Resize the filesystem

After the partition has been resized, let’s resize the filesystem so we can actually use it.

# Resize the FS
ubuntu@i-00cfd0679ab3871a3:~$ sudo resize2fs /dev/nvme0n1p1
...
Filesystem at /dev/nvme0n1p1 is mounted on /; on-line resizing required
The filesystem on /dev/nvme0n1p1 is now 2883323 (4k) blocks long.

# Verify the filesystem size again
# Now the FS size is matched with the partition size
ubuntu@i-00cfd0679ab3871a3:~$ df -Th
Filesystem      Type      Size  Used Avail Use% Mounted on
/dev/root       ext4       11G  1.7G  8.9G  17% /

Preparing the partition

Here I have a new block device /dev/sdb of 5GB and will create one partition on it.

# List all block devices
vagrant@vagrant:~$ lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0   64G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    2G  0 part /boot
└─sda3                      8:3    0   62G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   31G  0 lvm  /
sdb                         8:16   0    5G  0 disk 

# Create a new partition
vagrant@vagrant:~$ sudo fdisk /dev/sdb

# Let's see all the block devices again
# There should be /dev/sdb1 if that's the only partition
vagrant@vagrant:~$ lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
...
sdb                         8:16   0    5G  0 disk 
└─sdb1                      8:17   0    5G  0 part

Creating the filesystem

Then let’s create an ext4 filesystem on top of that new partition.

vagrant@vagrant:~$ sudo mkfs.ext4 /dev/sdb1 
...
Writing superblocks and filesystem accounting information: done 

# Let's create a /backup directory and try to mount that FS onto that
vagrant@vagrant:~$ sudo mkdir /backup
vagrant@vagrant:~$ sudo mount /dev/sdb1 /backup/

# Check what's inside the partition. It's empty ofc
vagrant@vagrant:~$ ls /backup/
lost+found

# Check the FS type, size, and the mount
vagrant@vagrant:~$ df -Th
Filesystem                        Type    Size  Used Avail Use% Mounted on
...
/dev/sdb1                         ext4    4.9G   24K  4.6G   1% /backup

Making it persistent between reboots

At this point if you reboot the server, the /backup won’t be mounted to the partition automatically. You need to make it persistent by adding a new entry into the /etc/fstab file.

vagrant@vagrant:~$ sudo nano /etc/fstab
...
# Let's mount /dev/sdb1 to /backup using the ext4 filesystem with standard options
# and don’t worry about backing it up or checking it during boot (it's the 0 0 at the end).
# Read more: https://www.redhat.com/en/blog/etc-fstab
/dev/sdb1 /backup ext4 defaults 0 0

Now you can safely reboot without worrying the disk mount is gone.