Deep-Dive: Understanding and Implementing API Security with JWT

Last year, we hit a wall at 10 million API requests per day. Our session-based authentication was crushing our Redis cluster, and horizontal scaling wasn't helping. We were burning through $8k monthly just on session storage, and our p95 latency had crept up to 450ms. My CTO Sarah pulled me aside one Friday afternoon: "We need to rethink this entire auth layer."

That's when I dove deep into JWT (JSON Web Tokens). Not the "here's a quick tutorial" kind of dive - I mean the "let's tear apart every security consideration, benchmark every implementation choice, and figure out what actually works at scale" kind of dive.

Here's what I learned after six months of building, breaking, and rebuilding our authentication system. This isn't theory from the RFC specs (though I read those too). This is what actually happened when we moved 50 million daily requests from session-based auth to JWT, including the three major security incidents we narrowly avoided and the performance optimizations that saved us $40k annually.

Why We Abandoned Sessions (And Why You Might Too)

Before I get into JWT implementation, let me explain why we made the switch. Our Node.js API was serving a React SPA and three mobile apps. We'd been using express-session with Redis as our session store - a pretty standard setup that worked fine until it didn't.

The breaking point came during a product launch. We'd anticipated 3x normal traffic and scaled our API servers accordingly. What we didn't anticipate was the Redis session store becoming a single point of failure. At around 2pm EST, our Redis cluster hit max connections (10,000), and every new login attempt started failing. Users who were already logged in? Fine. New users? Locked out completely.

My colleague Jake suggested we just add more Redis nodes, but that would've cost us another $3k/month and didn't solve the fundamental problem: every single API request required a Redis lookup. That's 10 million Redis queries daily just to validate sessions. Our Redis CPU was consistently above 70%, and we were paying for memory to store millions of session objects that were mostly just user IDs and timestamps.

The math was brutal:

• Average session size: ~2KB (user data, timestamps, CSRF tokens)
• Active sessions: ~500k concurrent users
• Memory required: ~1GB just for session data
• Redis cluster cost: $8k/month for high-availability setup
• Latency overhead: 15-25ms per request for Redis lookup

I started researching alternatives and kept coming back to JWT. The promise was compelling: stateless authentication that didn't require database lookups. But I was skeptical. Everyone in our Slack channels kept warning about JWT security issues, and the Hacker News threads I read were full of "JWT is dangerous" hot takes.

So I spent two weeks doing a deep dive. I read the RFC 7519 spec, studied real-world implementations from Auth0 and Firebase, and most importantly, I built a test harness to benchmark different approaches against our actual traffic patterns.

How JWT Actually Works (The Parts They Don't Explain Well)

Most JWT tutorials show you the three-part structure (header.payload.signature) and call it a day. That's like explaining how a car works by pointing at the wheels. Let me show you what's actually happening under the hood, because understanding this is critical for implementing JWT securely.

A JWT is essentially a cryptographically signed JSON object. When I first started working with them, I made the mistake of thinking they were encrypted. They're not. They're signed, which is completely different and understanding this distinction saved us from a major security vulnerability.

Here's a real JWT from our production system (with sensitive data redacted):

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiI3ODkxMjMiLCJlbWFpbCI6InVzZXJAZXhhbXBsZS5jb20iLCJyb2xlcyI6WyJ1c2VyIiwicHJlbWl1bSJdLCJpYXQiOjE3MDUwNTEyMDAsImV4cCI6MTcwNTA1NDgwMH0.signature_here

Let's decode each part. The first section (header) decodes to:

{
  "alg": "RS256",
  "typ": "JWT"
}

This tells us we're using RSA with SHA-256 for signing. We switched from HS256 (HMAC) to RS256 after I realized we needed to distribute public keys to multiple services without sharing the signing secret. More on that decision later.

The second section (payload) contains our actual claims:

{
  "userId": "789123",
  "email": "[email protected]",
  "roles": ["user", "premium"],
  "iat": 1705051200,
  "exp": 1705054800
}

Here's where I made my first major mistake. Initially, I was stuffing everything into the payload - user preferences, last login timestamps, feature flags, you name it. Our tokens ballooned to 3KB, and we started hitting HTTP header size limits (most servers cap at 8KB for all headers combined).

The wake-up call came when mobile clients started reporting intermittent 431 errors (Request Header Fields Too Large). I spent an entire day debugging before I realized our JWT was 3.2KB, and when combined with other headers (User-Agent, Accept, etc.), we were exceeding the default Nginx limit of 4KB for request headers.

I stripped our JWT payload down to the absolute essentials:

• User identifier (required for authorization)
• Core roles/permissions (needed for access control)
• Issued-at and expiration timestamps (required for security)
• Token ID (for revocation, which I'll explain later)

Final token size: 420 bytes. Problem solved.

The third section is the signature, which is where the real security magic happens. The signature is created by taking the encoded header and payload, concatenating them with a period, and signing that string with your private key:

const signature = sign(
  base64UrlEncode(header) + '.' + base64UrlEncode(payload),
  privateKey,
  'RS256'
);

This signature is what makes JWT tamper-proof. If someone modifies the payload (say, changing their role from "user" to "admin"), the signature verification fails because the signature was created from the original payload. They can't create a new valid signature without your private key.

Here's the critical part that took me a while to understand: the payload is NOT encrypted. Anyone can decode it using base64. I tested this myself:

$ echo "eyJ1c2VySWQiOiI3ODkxMjMiLCJlbWFpbCI6InVzZXJAZXhhbXBsZS5jb20ifQ" | base64 -d

Output:

{"userId":"789123","email":"[email protected]"}

This means you should NEVER put sensitive data in a JWT. No passwords, no credit card numbers, no social security numbers. We learned this when a security researcher emailed us pointing out that user email addresses were visible in our JWTs. Thankfully, emails weren't considered sensitive in our threat model, but it was a good reminder.

Our Implementation Journey (Three Attempts Before We Got It Right)

Let me walk you through how we actually implemented JWT authentication, including the two approaches that failed and what we finally settled on.

Attempt 1: The Naive Approach (Lasted 2 Weeks)

My first implementation was embarrassingly simple. I used the jsonwebtoken npm package and created tokens on login:

// auth.controller.js - First attempt (DON'T USE THIS)
const jwt = require('jsonwebtoken');

async function login(req, res) {
  const { email, password } = req.body;
  
  // Validate credentials
  const user = await User.findOne({ email });
  if (!user || !await user.comparePassword(password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }
  
  // Generate JWT
  const token = jwt.sign(
    { userId: user.id, email: user.email, roles: user.roles },
    process.env.JWT_SECRET,
    { expiresIn: '24h' }
  );
  
  res.json({ token });
}

This worked great in development. I felt like a genius. Then we deployed to staging, and reality hit.

The first problem: we were using HS256 (symmetric signing) with a single secret shared across all our services. This meant every service that needed to verify tokens also had access to the signing secret. Our payments service could theoretically forge tokens. Our analytics service could create admin tokens. This violated the principle of least privilege.

The second problem: 24-hour expiration. I thought this was user-friendly - users stay logged in for a full day. But when we discovered a compromised account (someone had phished a user's credentials), we had no way to invalidate their token. We had to wait 24 hours for it to expire naturally, or force a password reset which invalidated all tokens (and logged out legitimate users too).

The third problem: no refresh mechanism. When tokens expired after 24 hours, users were abruptly logged out mid-session. Our support tickets tripled.

This implementation lasted exactly two weeks before Sarah told me to "fix it properly."

Attempt 2: Adding Refresh Tokens (Lasted 3 Months)

For the second iteration, I implemented a proper refresh token pattern. The idea: issue short-lived access tokens (15 minutes) and long-lived refresh tokens (7 days). When the access token expires, use the refresh token to get a new one without requiring the user to log in again.

Here's what that looked like:

// auth.controller.js - Second attempt (better, but still had issues)
const jwt = require('jsonwebtoken');
const crypto = require('crypto');
const RefreshToken = require('./models/RefreshToken');

async function login(req, res) {
  const { email, password } = req.body;
  
  const user = await User.findOne({ email });
  if (!user || !await user.comparePassword(password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }
  
  // Generate short-lived access token
  const accessToken = jwt.sign(
    { 
      userId: user.id, 
      email: user.email, 
      roles: user.roles,
      tokenType: 'access'
    },
    process.env.JWT_SECRET,
    { expiresIn: '15m' }
  );
  
  // Generate refresh token
  const refreshTokenValue = crypto.randomBytes(40).toString('hex');
  
  // Store refresh token in database
  await RefreshToken.create({
    token: refreshTokenValue,
    userId: user.id,
    expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000), // 7 days
    userAgent: req.headers['user-agent'],
    ipAddress: req.ip
  });
  
  res.json({ 
    accessToken, 
    refreshToken: refreshTokenValue,
    expiresIn: 900 // 15 minutes in seconds
  });
}

async function refresh(req, res) {
  const { refreshToken } = req.body;
  
  // Verify refresh token exists and hasn't expired
  const tokenDoc = await RefreshToken.findOne({
    token: refreshToken,
    expiresAt: { $gt: new Date() }
  }).populate('userId');
  
  if (!tokenDoc) {
    return res.status(401).json({ error: 'Invalid refresh token' });
  }
  
  // Generate new access token
  const accessToken = jwt.sign(
    { 
      userId: tokenDoc.userId.id, 
      email: tokenDoc.userId.email, 
      roles: tokenDoc.userId.roles,
      tokenType: 'access'
    },
    process.env.JWT_SECRET,
    { expiresIn: '15m' }
  );
  
  res.json({ accessToken, expiresIn: 900 });
}

This was significantly better. We could now revoke refresh tokens from the database, giving us a way to forcibly log out users. The short-lived access tokens meant that even if one was compromised, it would only be valid for 15 minutes.

But we ran into new problems:

Problem 1: Database Load

Every token refresh hit the database. With 500k concurrent users and tokens expiring every 15 minutes, we were doing ~33k refresh requests per minute during peak hours. Our MongoDB cluster was getting hammered.

I added Redis caching for refresh tokens, which helped:

// Cached refresh token verification
async function verifyRefreshToken(token) {
  // Check Redis cache first
  const cached = await redis.get(`refresh:${token}`);
  if (cached) {
    return JSON.parse(cached);
  }
  
  // Fall back to database
  const tokenDoc = await RefreshToken.findOne({
    token,
    expiresAt: { $gt: new Date() }
  }).populate('userId');
  
  if (tokenDoc) {
    // Cache for 1 hour
    await redis.setex(`refresh:${token}`, 3600, JSON.stringify(tokenDoc));
  }
  
  return tokenDoc;
}

Problem 2: Mobile App Token Management

Our mobile developers were struggling with token refresh logic. They had to intercept 401 responses, call the refresh endpoint, retry the original request, and handle race conditions when multiple requests failed simultaneously.

Sarah's feedback: "This is too complicated. Can't we make it simpler?"

Problem 3: Still Using HS256

We were still using symmetric signing, which meant all services shared the same secret. I knew we needed to switch to asymmetric signing (RS256), but I was worried about the performance implications and the complexity of key distribution.

Attempt 3: The Production-Grade Solution (What We Use Today)

After three months of iteration, here's what we finally settled on. This is the system handling our 50M+ daily requests today:

Architecture Changes:

1. Switched to RS256 (asymmetric signing)
2. Implemented a dedicated auth service
3. Added token rotation for refresh tokens
4. Built a token revocation system using Redis
5. Implemented proper key rotation

Here's the core implementation:

// auth.service.js - Production implementation
const jwt = require('jsonwebtoken');
const crypto = require('crypto');
const fs = require('fs');
const Redis = require('ioredis');

class AuthService {
  constructor() {
    // Load RSA keys (generated offline and stored securely)
    this.privateKey = fs.readFileSync('./keys/private.pem', 'utf8');
    this.publicKey = fs.readFileSync('./keys/public.pem', 'utf8');
    
    this.redis = new Redis({
      host: process.env.REDIS_HOST,
      port: process.env.REDIS_PORT,
      password: process.env.REDIS_PASSWORD,
      // Connection pooling for better performance
      maxRetriesPerRequest: 3,
      enableReadyCheck: true,
      lazyConnect: false
    });
  }
  
  async generateTokenPair(user) {
    const tokenId = crypto.randomUUID();
    const refreshTokenId = crypto.randomUUID();
    
    // Access token - short lived, contains user info
    const accessToken = jwt.sign(
      {
        userId: user.id,
        email: user.email,
        roles: user.roles,
        jti: tokenId, // JWT ID for revocation
        tokenType: 'access'
      },
      this.privateKey,
      {
        algorithm: 'RS256',
        expiresIn: '15m',
        issuer: 'api.ourapp.com',
        audience: 'ourapp-services'
      }
    );
    
    // Refresh token - longer lived, minimal data
    const refreshToken = jwt.sign(
      {
        userId: user.id,
        jti: refreshTokenId,
        tokenType: 'refresh'
      },
      this.privateKey,
      {
        algorithm: 'RS256',
        expiresIn: '7d',
        issuer: 'api.ourapp.com',
        audience: 'ourapp-services'
      }
    );
    
    // Store refresh token metadata in Redis for revocation
    await this.redis.setex(
      `refresh:${refreshTokenId}`,
      7 * 24 * 60 * 60, // 7 days in seconds
      JSON.stringify({
        userId: user.id,
        createdAt: Date.now(),
        // Store device info for security monitoring
        userAgent: user.userAgent,
        ipAddress: user.ipAddress
      })
    );
    
    return { accessToken, refreshToken };
  }
  
  async refreshAccessToken(refreshToken) {
    try {
      // Verify refresh token signature
      const decoded = jwt.verify(refreshToken, this.publicKey, {
        algorithms: ['RS256'],
        issuer: 'api.ourapp.com',
        audience: 'ourapp-services'
      });
      
      // Check if token has been revoked
      const tokenData = await this.redis.get(`refresh:${decoded.jti}`);
      if (!tokenData) {
        throw new Error('Refresh token has been revoked');
      }
      
      // Fetch fresh user data (roles might have changed)
      const user = await User.findById(decoded.userId);
      if (!user) {
        throw new Error('User not found');
      }
      
      // Generate new token pair (rotation strategy)
      const newTokenPair = await this.generateTokenPair({
        id: user.id,
        email: user.email,
        roles: user.roles,
        userAgent: user.userAgent,
        ipAddress: user.ipAddress
      });
      
      // Revoke old refresh token
      await this.redis.del(`refresh:${decoded.jti}`);
      
      return newTokenPair;
    } catch (error) {
      if (error.name === 'TokenExpiredError') {
        throw new Error('Refresh token has expired');
      }
      if (error.name === 'JsonWebTokenError') {
        throw new Error('Invalid refresh token');
      }
      throw error;
    }
  }
  
  async revokeRefreshToken(tokenId) {
    // Immediately revoke a specific refresh token
    await this.redis.del(`refresh:${tokenId}`);
  }
  
  async revokeAllUserTokens(userId) {
    // Revoke all refresh tokens for a user (e.g., on password change)
    const keys = await this.redis.keys(`refresh:*`);
    const pipeline = this.redis.pipeline();
    
    for (const key of keys) {
      const data = await this.redis.get(key);
      if (data) {
        const parsed = JSON.parse(data);
        if (parsed.userId === userId) {
          pipeline.del(key);
        }
      }
    }
    
    await pipeline.exec();
  }
  
  verifyAccessToken(token) {
    try {
      const decoded = jwt.verify(token, this.publicKey, {
        algorithms: ['RS256'],
        issuer: 'api.ourapp.com',
        audience: 'ourapp-services'
      });
      
      // Additional validation
      if (decoded.tokenType !== 'access') {
        throw new Error('Invalid token type');
      }
      
      return decoded;
    } catch (error) {
      if (error.name === 'TokenExpiredError') {
        throw new Error('Access token has expired');
      }
      if (error.name === 'JsonWebTokenError') {
        throw new Error('Invalid access token');
      }
      throw error;
    }
  }
}

module.exports = new AuthService();

This implementation solved all our previous problems:

RS256 Benefits:

• Services only need the public key to verify tokens
• Only the auth service has the private key for signing
• We can distribute the public key freely without security concerns
• Better separation of concerns

Token Rotation:

• Every refresh generates a new refresh token
• Old refresh token is immediately revoked
• Prevents refresh token reuse attacks
• Limits the damage if a refresh token is stolen

Redis for Revocation:

• Fast lookups (sub-millisecond)
• TTL automatically expires old tokens
• Scales horizontally
• Much cheaper than database queries

Here's the middleware we use to protect routes:

// middleware/auth.middleware.js
const authService = require('../services/auth.service');

function requireAuth(req, res, next) {
  const authHeader = req.headers.authorization;
  
  if (!authHeader || !authHeader.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'No token provided' });
  }
  
  const token = authHeader.substring(7); // Remove 'Bearer ' prefix
  
  try {
    const decoded = authService.verifyAccessToken(token);
    req.user = decoded;
    next();
  } catch (error) {
    return res.status(401).json({ error: error.message });
  }
}

// Role-based access control
function requireRole(...allowedRoles) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({ error: 'Not authenticated' });
    }
    
    const hasRole = req.user.roles.some(role => allowedRoles.includes(role));
    if (!hasRole) {
      return res.status(403).json({ error: 'Insufficient permissions' });
    }
    
    next();
  };
}

module.exports = { requireAuth, requireRole };

Usage in routes:

// routes/api.routes.js
const express = require('express');
const { requireAuth, requireRole } = require('../middleware/auth.middleware');

const router = express.Router();

// Public route
router.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});

// Protected route - any authenticated user
router.get('/profile', requireAuth, async (req, res) => {
  const user = await User.findById(req.user.userId);
  res.json(user);
});

// Admin-only route
router.delete('/users/:id', requireAuth, requireRole('admin'), async (req, res) => {
  await User.findByIdAndDelete(req.params.id);
  res.json({ success: true });
});

// Multiple roles allowed
router.post('/content', requireAuth, requireRole('editor', 'admin'), async (req, res) => {
  // Create content
});

module.exports = router;

The Security Gotchas We Discovered (Learn From Our Mistakes)

Let me share the security issues we encountered and how we fixed them. Some of these were caught in code review, others we discovered in production (thankfully before they were exploited).

Gotcha 1: Algorithm Confusion Attack

This one almost got us. The JWT spec allows the client to specify the algorithm in the header. An attacker can change the algorithm from RS256 to HS256, and if your verification code isn't careful, it might use the public key as an HMAC secret.

Here's what the attack looks like:

// Attacker modifies the JWT header
{
  "alg": "HS256",  // Changed from RS256
  "typ": "JWT"
}

Then they sign the token using your public key (which is public!) as the HMAC secret. If your verification code doesn't explicitly check the algorithm, it might accept this forged token.

We caught this during a security audit. Our original verification code was vulnerable:

// VULNERABLE CODE - DO NOT USE
function verifyToken(token) {
  return jwt.verify(token, publicKey); // Doesn't specify algorithm!
}

The fix was simple but critical:

// SECURE CODE
function verifyToken(token) {
  return jwt.verify(token, publicKey, {
    algorithms: ['RS256'] // Explicitly whitelist allowed algorithms
  });
}

Always specify the algorithms parameter. Never trust the algorithm specified in the token header.

Gotcha 2: Missing Token Expiration Validation

I assumed the jsonwebtoken library would reject expired tokens automatically. It does, but only if you actually check for the error properly. We had code that looked like this:

// PROBLEMATIC CODE
async function someProtectedRoute(req, res) {
  try {
    const decoded = jwt.verify(token, publicKey, { algorithms: ['RS256'] });
    // Do stuff with decoded token
  } catch (error) {
    console.log('Token error:', error); // Just logging, not handling!
    // Code continues executing...
  }
}

The catch block was logging errors but not actually returning or throwing. Expired tokens were being logged but the request continued processing. We discovered this when a penetration tester showed us they could use expired tokens to access protected resources.

The fix:

// CORRECT CODE
async function someProtectedRoute(req, res) {
  try {
    const decoded = jwt.verify(token, publicKey, { algorithms: ['RS256'] });
    // Do stuff with decoded token
  } catch (error) {
    if (error.name === 'TokenExpiredError') {
      return res.status(401).json({ error: 'Token expired' });
    }
    if (error.name === 'JsonWebTokenError') {
      return res.status(401).json({ error: 'Invalid token' });
    }
    // Log unexpected errors but still reject the request
    console.error('Unexpected token verification error:', error);
    return res.status(401).json({ error: 'Authentication failed' });
  }
}

Gotcha 3: Sensitive Data in Payload

I mentioned this earlier, but it's worth emphasizing because we made this mistake multiple times. Remember: JWT payloads are BASE64-encoded, not encrypted. Anyone can decode them.

We initially included user email addresses, which seemed harmless. Then we added phone numbers for our 2FA feature. Then someone added a "lastPasswordChange" timestamp. Before we knew it, we were leaking user metadata.

The wake-up call came when a security researcher showed us they could decode our JWTs and enumerate active users, see which users had premium accounts (we'd included a "subscriptionTier" field), and even identify recently created accounts (from the "iat" timestamp combined with user ID patterns).

Our rule now: Only include data in the JWT that you'd be comfortable displaying publicly. Everything else should be fetched from the database after token verification.

// MINIMAL JWT PAYLOAD - What we use now
{
  "userId": "789123",           // Required for identification
  "roles": ["user", "premium"], // Required for authorization
  "jti": "uuid-here",          // Required for revocation
  "iat": 1705051200,           // Required - issued at
  "exp": 1705054800            // Required - expiration
}

Gotcha 4: Token Revocation Challenges

This was our biggest architectural challenge. JWTs are stateless by design, which means once issued, they're valid until they expire.