Learning Istio the Hard Way: A Real Service Mesh Lab with Canary, mTLS, and Tracing.

Source: Dev.to

Why I Built a Service Mesh Lab Instead of Just Reading Docs

This project started as a personal lab to really understand what a service mesh does beyond the buzzwords. Instead of using sample apps, the goal was to take a real 3‑tier app (Next.js frontend, Go backend, Flask ML service) and see how Istio changes the way traffic, security, and observability work in practice.

The idea was simple: if this setup feels like something that could ship to production one day, then the learning will stick and this repo becomes a living reference for future work.

The 3RVision Platform: Real App, Real Traffic

3RVision is split into three logical services, each running in its own Kubernetes namespace:

• frontend → Next.js UI
• backend → Go API server
• ml → Flask ML inference service

The frontend talks to the backend, and the backend calls the ML service for model inference – exactly the kind of hop‑by‑hop traffic that benefits from a service mesh.

Each service has two deployment variants:

• stable version (production)
• canary version (testing new features)

This is where Istio’s traffic‑management features come into play.

Setting the Stage: Kind + Terraform + Namespaces

To avoid dealing with cloud accounts, Terraform provisions a local Kind cluster with:

1. 1 control‑plane node
2. 2 worker nodes
3. Port mappings for HTTP (80) and HTTPS (443)

Cluster Setup Workflow

# Provision the cluster
terraform init
terraform apply

# Create namespaces
kubectl create namespace frontend
kubectl create namespace backend
kubectl create namespace ml

# Enable Istio sidecar injection
kubectl label namespace frontend istio-injection=enabled
kubectl label namespace backend istio-injection=enabled
kubectl label namespace ml istio-injection=enabled

This gives you a clean separation:
Terraform → cluster lifecycle
Kubernetes → application resources
Istio → traffic shaping and security

How Istio Fits In: Sidecars, Gateways, and the Data Plane

Istio works by injecting an Envoy sidecar proxy next to each application container. All inbound and outbound traffic flows through this sidecar, which means you can add routing, retries, mTLS, and telemetry without changing application code.

Architecture Overview

                            ┌─────────────────────┐
                            │    User/Client      │
                            └──────────┬──────────┘
                                       │
                                       ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                        KIND KUBERNETES CLUSTER                            │
│                        (Terraform Provisioned)                            │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐              │
│  │ Control Plane  │  │   Worker #1    │  │   Worker #2    │              │
│  └────────────────┘  └────────────────┘  └────────────────┘              │
├──────────────────────────────────────────────────────────────────────────┤
│                          ISTIO SERVICE MESH                               │
│                                                                           │
│    Gateway ──────► VirtualService ──────► DestinationRule                │
│   (Ingress)          (Routing)           (mTLS + Load Balancing)         │
├──────────────────────────────────────────────────────────────────────────┤
│                           MICROSERVICES                                   │
│                                                                           │
│   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐               │
│   │   FRONTEND   │    │   BACKEND    │    │   ML MODEL   │               │
│   │   (Next.js)  │───►│     (Go)     │───►│   (Flask)    │               │
│   │  Port: 3000  │    │  Port: 8080  │    │  Port: 5001  │               │
│   │              │    │              │    │              │               │
│   │ stable/canary│    │ stable/canary│    │ stable/canary│               │
│   └──────────────┘    └──────────────┘    └──────────────┘               │
├──────────────────────────────────────────────────────────────────────────┤
│                        OBSERVABILITY STACK                                │
│                                                                           │
│   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐               │
│   │  Prometheus  │    │    Jaeger    │    │   Grafana    │               │
│   │   (Metrics)  │    │  (Tracing)   │    │ (Dashboards) │               │
│   │  Port: 9090  │    │ Port: 16686  │    │  Port: 3000  │               │
│   └──────────────┘    └──────────────┘    └──────────────┘               │
└──────────────────────────────────────────────────────────────────────────┘

At the edge, an Istio Ingress Gateway receives external requests, applies routing rules defined by VirtualServices, and forwards traffic deeper into the mesh.

Traffic Management 101: VirtualServices, DestinationRules, and Subsets

The main Istio building blocks used in this project are:

Example: Frontend VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: frontend-vs
  namespace: frontend
spec:
  hosts:
    - "frontend.local"
  gateways:
    - frontend-gateway
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: frontend-service
            subset: canary
          weight: 100
    - route:
        - destination:
            host: frontend-service
            subset: stable
          weight: 100