Source: Dev.to

Starting a New Node.js Project – From Zero to Production‑Ready

Starting a new Node.js project often involves tedious repetitive tasks: scaffolding directory structures, setting up Express, configuring database connections, managing migrations, and integrating messaging systems like Kafka. This boilerplate phase can eat up hours of your initial development time.

Today, I’ll show you how to go from zero to a production‑ready environment in minutes. We will build a high‑performance Node.js service using:

• Clean Architecture
• TypeScript
• MySQL
• Flyway for database migrations
• Kafka for real‑time event‑driven messaging
• Docker Compose for orchestration
• GitHub Actions for CI/CD

Let’s dive in!

🎯 Ready‑to‑Run Source Code

Instead of manually copying snippets, I’ve packaged the entire source code for this article into a production‑grade template on GitHub. The project already has 3,000+ downloads and is being used by developers for real‑world services.

Repository
paudang/nodejs-clean-rest-kafka

Quick start

git clone https://github.com/paudang/nodejs-clean-rest-kafka.git
cd nodejs-clean-rest-kafka
docker-compose up -d

(Just run the commands above and you’re live!)

Step 1 – Initialize Project & Install Dependencies

# Create project folder
mkdir nodejs-clean-rest-kafka
cd nodejs-clean-rest-kafka
npm init -y

Install production libraries

npm install express cors helmet hpp express-rate-limit dotenv morgan kafkajs sequelize mysql2 winston

Install development dependencies

npm install -D typescript @types/node @types/express @types/cors @types/morgan \
  ts-node tsconfig-paths tsc-alias jest ts-jest @types/jest

Initialise tsconfig.json with path aliases (@/*)

{
  "compilerOptions": {
    "target": "es2020",
    "module": "commonjs",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  },
  "include": ["src/**/*"]
}

Step 2 – Architecting for Scale (Clean Architecture) 🏗️

Using Clean Architecture keeps the codebase decoupled and highly testable.
Create the folder structure for the different layers:

mkdir -p src/domain src/usecases src/interfaces src/infrastructure src/utils

This separation lets you swap databases, messaging systems, or web frameworks without touching the core business logic.

Step 3 – Event‑Driven Messaging with Kafka 🚀

In a microservices architecture, Kafka acts as the “heartbeat” for asynchronous communication.
Below is a KafkaService that uses the Connection‑Promise pattern to guarantee the producer is fully connected before any messages are sent, preventing data loss during startup.

src/infrastructure/messaging/kafkaClient.ts

import { Kafka, Producer } from 'kafkajs';

export class KafkaService {
  private producer: Producer;
  private isConnected = false;
  private connectionPromise: Promise<void> | null = null;

  constructor() {
    const kafka = new Kafka({
      clientId: 'user-service',
      brokers: ['localhost:9092'],
    });

    this.producer = kafka.producer();
  }

  /** 
   * Returns a promise that resolves once the producer is connected.
   * Subsequent calls reuse the same promise, ensuring a single connection attempt.
   */
  async connect(): Promise<void> {
    if (this.connectionPromise) return this.connectionPromise;

    this.connectionPromise = (async () => {
      await this.producer.connect();
      this.isConnected = true;
      console.log('[Kafka] Producer connected successfully');
    })();

    return this.connectionPromise;
  }

  /**
   * Sends an event to the specified Kafka topic.
   *
   * @param topic   - Kafka topic name.
   * @param action  - Identifier for the type of event (e.g., "UserCreated").
   * @param payload - Event payload; will be JSON‑stringified.
   */
  async sendEvent(topic: string, action: string, payload: unknown): Promise<void> {
    await this.connect(); // Ensure the producer is ready

    await this.producer.send({
      topic,
      messages: [
        {
          value: JSON.stringify({
            action,
            payload,
            ts: new Date().toISOString(),
          }),
        },
      ],
    });

    console.log(`[Kafka] Triggered ${action} → ${topic}`);
  }
}

/** Singleton instance for easy import throughout the codebase */
export const kafkaService = new KafkaService();

Key points

• Connection‑Promise pattern guarantees a single, reusable connection attempt.
• sendEvent always awaits connect() before publishing, eliminating race conditions on startup.
• The service is exported as a singleton (kafkaService) for convenient reuse across microservices.

Step 3.5 – Scaling Consumers with Clean Interfaces 🛠️

Managing dozens of event types can quickly become messy.
We use abstract base classes and schema validation to keep consumers organized.

1️⃣ BaseConsumer – The Blueprint

File: src/interfaces/messaging/baseConsumer.ts

import { EachMessagePayload } from 'kafkajs';

/**
 * Abstract base class for all Kafka consumers.
 * Provides a common entry‑point (`onMessage`) that parses the raw
 * payload and forwards the result to the concrete `handle` method.
 */
export abstract class BaseConsumer {
  /** Topic this consumer subscribes to */
  abstract topic: string;

  /** Business logic for handling a parsed message */
  abstract handle(data: unknown): Promise<void>;

  /** Common message handling – parses JSON and forwards to `handle` */
  async onMessage({ message }: EachMessagePayload): Promise<void> {
    const rawValue = message.value?.toString();
    if (!rawValue) return;

    const data = JSON.parse(rawValue);
    await this.handle(data);
  }
}

2️⃣ Schema Validation – The Contract

We use Zod to define the contract between producer and consumer.
(Example schema omitted for brevity – see src/interfaces/messaging/schemas/userEventSchema.ts.)

3️⃣ WelcomeEmailConsumer – The Implementation

File: src/interfaces/messaging/welcomeEmailConsumer.ts

import { BaseConsumer } from '../baseConsumer';
import { userCreatedSchema } from './schemas/userEventSchema';
import { sendWelcomeEmail } from '../../utils/email';

export class WelcomeEmailConsumer extends BaseConsumer {
  /** Topic this consumer listens to */
  topic = 'user-topic';

  /** Validate payload and trigger the welcome‑email flow */
  async handle(data: unknown): Promise<void> {
    // Validate payload against the Zod schema
    const parsed = userCreatedSchema.parse(data);

    await sendWelcomeEmail(parsed.email, parsed.name);
    console.log(`[Consumer] Sent welcome email to ${parsed.email}`);
  }
}

TL;DR

• BaseConsumer – centralises JSON parsing & error handling.
• Zod schemas – enforce a strict contract for each event type.
• Concrete consumers (e.g., WelcomeEmailConsumer) – focus solely on business logic.

This pattern scales cleanly as the number of event types grows.

What’s Next?

• Database migrations with Flyway (add flyway.conf and migration scripts).
• CI/CD pipelines using GitHub Actions (build, test, Docker image push).
• Testing: unit tests for use‑cases, integration tests for controllers, and contract tests for Kafka messages.

Feel free to explore the full template in the linked repository and adapt it to your own domain! 🚀

const result = UserEventSchema.safeParse(data);
if (result.success && result.data.action === 'USER_CREATED') {
  console.log(`[Kafka] 📧 Sending welcome email to ${result.data.payload.email}...`);
}

Step 4 – Database Version Control with Flyway

Manual database changes are a nightmare in production. Flyway solves this problem by managing schema versioning through simple, incremental SQL migration files.

Example migration file

V1__Create_Users_Table.sql

CREATE TABLE users (
    id          INT AUTO_INCREMENT PRIMARY KEY,
    name        VARCHAR(255) NOT NULL,
    email       VARCHAR(255) NOT NULL UNIQUE,
    created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

• The V1 prefix denotes the version number.
• The double‑underscore (__) separates the version from the description.
• Each migration file is applied in order, ensuring the database schema stays in sync across environments.

Step 5 – Clean Use‑Cases & Controllers

1️⃣ Use‑Case (Interactor)

The business logic is isolated in src/usecases/createUser.ts:

export const createUserUseCase = async (name: string, email: string) => {
  // Persist the user via the repository abstraction
  const user = await userRepository.save({ name, email });

  // Emit a domain event
  await kafkaService.sendEvent(
    'user-events',
    'USER_CREATED',
    { id: user.id, email: user.email }
  );

  return user;
};

2️⃣ Controller (Interface Layer)

The controller only handles the HTTP request/response cycle and delegates to the use‑case:

import { Request, Response } from 'express';
import { createUserUseCase } from '@/usecases/createUser';

export const createUser = async (req: Request, res: Response) => {
  try {
    const { name, email } = req.body;
    const user = await createUserUseCase(name, email);
    res.status(201).json(user);
  } catch (err) {
    // TODO: replace with proper error handling / logging
    res.status(500).json({ error: 'Internal Server Error' });
  }
};

The controller stays thin, while the use‑case contains all the domain‑specific work.

Step 6 – Docker for Production Excellence

1. Dockerfile (Multi‑stage build)

Separate the build environment from the runtime to keep the final image small and reduce the attack surface:

# ---------- Builder stage ----------
FROM node:22-alpine AS builder
WORKDIR /app

# Install dependencies needed for the build
COPY package*.json ./
RUN npm ci

# Copy source files and build the app
COPY . .
RUN npm run build


# ---------- Production stage ----------
FROM node:22-alpine AS production
WORKDIR /app
ENV NODE_ENV=production
ENV NPM_CONFIG_UPDATE_NOTIFIER=false

# Install only production dependencies
COPY package*.json ./
RUN npm ci --only=production

# Pull the compiled output from the builder stage
COPY --from=builder /app/dist ./dist

EXPOSE 3000
CMD ["npm", "start"]

2. docker‑compose.yml (Full‑stack stack)

One command to bring up the whole system:

version: "3.9"

services:
  app:
    build: .
    ports:
      - "3000:3000"
    depends_on:
      - db
      - kafka
    environment:
      - KAFKA_BROKER=kafka:29092
      - DB_HOST=db

  db:
    image: mysql:8.0
    ports:
      - "3306:3306"
    volumes:
      - ./flyway/sql:/docker-entrypoint-initdb.d
    environment:
      - MYSQL_ROOT_PASSWORD=example   # adjust as needed
      - MYSQL_DATABASE=myapp

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1

  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    ports:
      - "2181:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

All services are defined in a single docker‑compose.yml file, allowing you to start the entire stack with docker compose up -d.

One Last Surprise… 🤫

Think this took hours to set up?

The Truth Is: The entire project structure—Clean Architecture, Kafka Producers/Consumers, Flyway migrations, Docker configs, and CI/CD pipelines—was generated in under 60 seconds using an automation tool I built.

Time‑to‑market is everything. Stop reinventing the wheel and start shipping business value.

Want to generate a “perfect” Node.js repo like this yourself? Try my CLI tool:

npx nodejs-quickstart-structure init

🔗 Full Documentation: nodejs-quickstart-structure

Building production‑ready software shouldn’t be a chore. I hope this helps you ship your next big idea faster than ever. If you found this useful, don’t forget to give a Star ⭐ on GitHub! 🔥