HLD: Social Media System (Twitter / Instagram)

Understanding the Problem

What is a Social Media System?

A social media platform lets users create and share content (posts, images, videos), follow other users, and consume a personalized feed of content from people they follow. The central engineering challenge is generating a news feed that feels real-time for hundreds of millions of users, especially when some users (celebrities) have tens of millions of followers, making naive fan-out approaches collapse under load.

Functional Requirements

Core (above the line):

1. Create a post -- users publish text, images, or short videos
2. Follow/unfollow users -- users can follow other users to see their posts
3. News feed -- a personalized, chronologically (or rank-ordered) feed of posts from followed users
4. View user profile -- see a user's posts, follower count, and following count

Below the line (mention but don't design):

• Like, comment, share/retweet
• Direct messaging
• Hashtags and trending topics
• Content moderation
• Stories/ephemeral content
• Push notifications

Non-Functional Requirements

1. Feed freshness -- new posts appear in followers' feeds within 5 seconds for most users
2. Low latency -- feed loads in < 200ms (P99)
3. High availability -- 99.99% uptime
4. Scale -- 500M DAU, 200M posts/day, average user follows 200 people, top 1% of users (celebrities) have > 1M followers

---

The Set Up

Core Entities

Entity	Description
User	id, username, displayName, bio, followerCount, followingCount, isVerified
Post	id, authorId, content, mediaUrls, createdAt, likeCount, commentCount
Follow	followerId, followeeId, createdAt
FeedItem	userId, postId, authorId, score, createdAt (denormalized for fast reads)

API Design

Create a post:

POST /api/posts
Authorization: Bearer <token>

Request:
{
  "content": "Just shipped a new feature!",
  "mediaUrls": ["https://cdn.example.com/img/abc.jpg"]
}

Response: 201 Created
{
  "postId": "post_abc123",
  "authorId": "user_456",
  "createdAt": "2024-01-15T10:30:00Z"
}

Get news feed (paginated):

GET /api/feed?cursor=<score_or_timestamp>&limit=20
Authorization: Bearer <token>

Response: 200 OK
{
  "posts": [
    {
      "postId": "post_abc123",
      "author": { "id": "user_456", "username": "janedoe", "avatar": "..." },
      "content": "Just shipped a new feature!",
      "mediaUrls": ["..."],
      "createdAt": "2024-01-15T10:30:00Z",
      "likeCount": 142,
      "commentCount": 23,
      "liked": false
    }
  ],
  "nextCursor": "1705312000_post_xyz",
  "hasMore": true
}

Follow a user:

POST /api/users/{userId}/follow
Authorization: Bearer <token>

Response: 200 OK
{
  "following": true,
  "followeeId": "user_789"
}

Get user profile:

GET /api/users/{userId}

Response: 200 OK
{
  "id": "user_789",
  "username": "elonmusk",
  "followerCount": 150000000,
  "followingCount": 450,
  "postCount": 32000,
  "bio": "..."
}

Get user's posts:

GET /api/users/{userId}/posts?cursor=<timestamp>&limit=20

---

High-Level Design

Flow 1: Creating a Post (Fan-Out on Write for Normal Users)

[Client] --> [API Gateway] --> [Post Service] --> [PostgreSQL (posts table)]
                                    |
                                    v
                              [Kafka: post_created]
                                    |
                                    v
                              [Fan-Out Service] --> [Redis: user feed caches]
                                    |
                              [Follower Service] (get follower list)

Step-by-step for a normal user (< 10K followers):

1. User creates a post. Post Service validates and writes to PostgreSQL (posts table, partitioned by authorId)
2. Post Service publishes a PostCreated event to Kafka, partitioned by authorId
3. Fan-Out Service consumes the event. It queries the Follower Service (backed by a separate datastore) to get the list of followers for this author
4. For each follower, Fan-Out Service prepends the postId to the follower's feed cache in Redis:

   LPUSH feed:user_follower1 "post_abc123"
   LTRIM feed:user_follower1 0 799  -- keep only last 800 items

5. The feed cache now has the new post ready for when each follower opens their feed

Back-of-envelope for fan-out:

• 200M posts/day = ~2,300 posts/sec
• Average user has 200 followers
• Fan-out operations: 2,300 * 200 = 460K Redis writes/sec
• Redis handles ~200K ops/sec per shard, so we need ~3 Redis shards just for fan-out writes. Very manageable.

Flow 2: Reading the News Feed

1. User opens the app. Client calls GET /api/feed?limit=20
2. Feed Service reads the user's feed from Redis: LRANGE feed:user_123 0 19
3. This returns a list of postIds. Feed Service does a multi-get from the Post Cache (Redis) to fetch the full post objects
4. For cache misses, Feed Service falls back to PostgreSQL
5. Feed Service hydrates author info from the User Cache (Redis) and returns the assembled feed
6. Client renders the feed

Latency breakdown:

• Redis LRANGE: ~1ms
• Redis MGET for 20 posts: ~2ms
• User info hydration: ~2ms (cached)
• Total: ~5ms server-side. Well under our 200ms target.

Flow 3: Following a New User

1. Client calls POST /api/users/{userId}/follow
2. Follow Service writes the relationship to the Social Graph store (PostgreSQL or a graph database like DGraph for complex queries)
3. Follow Service increments followerCount for the followee and followingCount for the follower (can be done async via Kafka for consistency)
4. Follow Service triggers a feed backfill: fetch the last 20 posts from the new followee and merge them into the follower's feed cache
5. Publish UserFollowed event to Kafka for analytics and recommendations

Flow 4: Celebrity Posts (Hybrid Fan-Out)

For a celebrity with 50M followers, fan-out on write is not feasible: writing to 50M Redis keys per post would take 50M / 200K ops/sec = 250 seconds per post. That violates our 5-second freshness requirement.

Solution: Fan-out on read for celebrities.

1. Celebrity creates a post. It is written to PostgreSQL and stored in a Celebrity Post Cache (Redis sorted set per celebrity, sorted by timestamp)
2. No fan-out happens -- the post is NOT pushed to followers' feed caches
3. When a follower opens their feed: a. Feed Service reads their precomputed feed from Redis (posts from normal users they follow) b. Feed Service also checks the Celebrity Post Caches for each celebrity they follow (typically < 20 celebrities per user) c. Feed Service merges these two lists in memory, sorted by timestamp/score d. Returns the merged feed to the client

---

Potential Deep Dives

Deep Dive 1: Fan-Out on Write vs Fan-Out on Read (Hybrid Approach)

Bad Solution -- Pure fan-out on write: Every post is pushed to every follower's feed. Celebrity with 50M followers posts 10 times/day = 500M writes/day just for one user. Total across all celebrities: if 5M users have > 10K followers, the write amplification is catastrophic.

Back-of-envelope: 5M "popular" users * 10 posts/day * 100K avg followers = 5 trillion fan-out operations/day. At 200K ops/sec per Redis shard, we would need 290,000 Redis shards. Absurd.

Good Solution -- Pure fan-out on read: No precomputation. When a user opens their feed, query all followed users' recent posts and merge. User follows 200 people: 200 queries, even cached, is 200 * 1ms = 200ms. That is at our P99 limit and leaves no room for ranking or hydration. Also, this hits the read path hard -- 500M DAU * 10 feed loads/day = 5B feed queries/day.

Great Solution -- Hybrid approach (what Twitter actually uses):

Classify users into two tiers:

• Normal users (< 10K followers): fan-out on write. Their posts are pushed to followers' feed caches.
• Celebrity users (> 10K followers): fan-out on read. Their posts are stored separately and merged at read time.

Fan-Out Decision Logic:
if author.followerCount < CELEBRITY_THRESHOLD:
    fanOutOnWrite(post, author.followers)
else:
    addToCelebrityCache(post, author.id)

Threshold tuning: The threshold is typically 10K-100K. The sweet spot depends on:

• Redis cluster capacity for writes (fan-out on write cost)
• Average number of celebrities a user follows (fan-out on read cost)
• Acceptable feed generation latency

If average user follows 15 celebrities, the read-time merge adds ~15ms. Acceptable.

Deep Dive 2: News Feed Generation and Ranking

Bad Solution -- Pure chronological: Show posts in reverse chronological order. Simple, but users miss important posts when they follow many accounts. Engagement drops because high-quality posts from hours ago are buried under recent low-quality posts.

Good Solution -- Simple ranking with feature-based scoring: Score each post with a formula:

score = (recency_weight * recency) +
        (engagement_weight * (likes + 2*comments + 3*shares)) +
        (affinity_weight * user_author_interaction_score)

Where user_author_interaction_score is based on how often the user has liked/commented on this author's posts in the past 30 days.

Store the score in the feed cache: ZADD feed:user_123 <score> <postId>. Retrieve with ZREVRANGE.

Great Solution -- ML-based ranking pipeline:

1. Candidate generation: Pull ~500 candidates from the feed cache (fan-out results + celebrity cache merge)
2. Feature extraction: For each candidate, extract features:
   • Post features: age, media type, engagement rate, content category
   • Author features: author-user affinity, posting frequency
   • User features: preferred content types, active hours, past engagement patterns
3. Ranking model: A lightweight ML model (e.g., gradient-boosted trees or a small neural network) scores each post. The model is trained offline on engagement data (clicks, likes, dwell time).
4. Re-ranking: Apply diversity rules (don't show 5 posts from the same author in a row), content policy filters, and ad insertion.
5. Return top 20.

Latency budget: This must complete in < 100ms. The ML model inference on 500 candidates typically takes ~20ms on a GPU inference server. Feature extraction is the bottleneck -- precompute and cache features.

Deep Dive 3: Hot User / Viral Content Detection

The Problem: A previously normal user goes viral. They had 5K followers, but a post gets shared and suddenly 2M people visit their profile in an hour. The fan-out on write already happened for their 5K followers, but now 2M non-followers are viewing the post.

Bad Solution -- No special handling: 2M users load the profile page, each hitting the database. PostgreSQL collapses under load for this single user.

Good Solution -- Read-through cache with short TTL: Cache the user's profile and recent posts in Redis with a 60-second TTL. First request after TTL misses and reloads from DB, subsequent requests are served from cache. Works for moderate traffic.

Great Solution -- Adaptive caching with hot-spot detection:

1. Hot-spot detector: A lightweight service monitors request rates per entity (user, post). Uses a Count-Min Sketch or Redis HyperLogLog to approximate request frequency with O(1) space.
2. When an entity exceeds a threshold (e.g., > 1K requests/minute), it is flagged as "hot."
3. Hot entities are:
   • Promoted to a dedicated cache tier (separate Redis cluster for hot data, with longer TTL and higher replication)
   • Served from a CDN edge cache if the content is static (profile page, image)
   • Rate-limited on the write side (prevent cache stampede on update)
4. When the traffic dies down, the entity is demoted back to normal caching.

Implementation pattern -- Leaky bucket for hot detection:

# In-memory counter per entity, decays over time
hot_counter = defaultdict(float)

def on_request(entity_id):
    hot_counter[entity_id] += 1
    if hot_counter[entity_id] > HOT_THRESHOLD:
        promote_to_hot_cache(entity_id)

# Background thread: decay all counters every 10 seconds
def decay():
    for key in hot_counter:
        hot_counter[key] *= 0.9  # exponential decay

Deep Dive 4: Caching Strategy for Viral Content

The Problem: A single viral post gets 10M views in an hour. Without caching, that is 10M DB reads for one row.

Bad Solution -- No caching: 10M reads/hour = 2,778 QPS for one post. The DB is fine for one post, but if 100 posts go viral simultaneously, that is 278K QPS on the posts table. PostgreSQL struggles.

Good Solution -- Cache-aside with stale-while-revalidate:

1. Check Redis for post data
2. If hit: return cached data
3. If miss: fetch from DB, write to Redis with TTL = 5 minutes, return
4. For engagement counters (likes, comments): use Redis INCR for real-time counts,
   periodically flush to DB (every 30 seconds)

The "stale-while-revalidate" pattern: when TTL is about to expire, one request triggers an async background refresh while serving the stale data. Prevents thundering herd.

Great Solution -- Write-through with separate counter caching:

Separate immutable post data from mutable counters:

• Post content (text, media URLs, author): cache in Redis with long TTL (1 hour). Content does not change.
• Engagement counters (likes, comments, shares): stored in Redis as atomic counters (INCR post:abc123:likes). Periodically flushed to the database. Never cached with TTL -- always live values.
• User-specific state (did I like this post?): stored in a separate Redis set per user, or computed from the likes table with bloom filters for fast negative lookups.

CDN for media: All images and videos are served from CloudFront or similar CDN. Even if a post goes viral, media delivery is handled by the CDN, not our servers. The origin server only handles the metadata.

Deep Dive 5: Social Graph Storage

The Problem: "Who does User A follow?" and "Who follows User A?" are the two most critical queries. With 500M users and average 200 followings each, the follow table has 100 billion rows.

Bad Solution -- Single PostgreSQL table:

CREATE TABLE follows (follower_id BIGINT, followee_id BIGINT, created_at TIMESTAMP);

100B rows in one table. Even with indexes, queries slow down. Sharding by follower_id makes "who follows User X?" expensive (scatter-gather across all shards).

Good Solution -- Dual-table approach with sharding: Maintain two tables:

following(userId, followeeId)  -- sharded by userId, answers "who do I follow?"
followers(userId, followerId)  -- sharded by userId, answers "who follows me?"

Write to both on follow/unfollow (via Kafka for eventual consistency). Each query only hits one shard.

Great Solution -- Graph database or adjacency list in Redis: For real-time fan-out, we need the follower list fast. Store it in Redis:

SADD followers:user_456 user_1 user_2 user_3 ...
SADD following:user_123 user_456 user_789 ...

Back-of-envelope: 500M users * 200 followings * 8 bytes per userId = 800 GB. This fits in a Redis cluster with ~30 shards (32GB each). For celebrities with millions of followers, store their follower list in the database (too large for Redis) and page through it during fan-out.

Hybrid: Redis for users with < 100K followers (fast read), database for celebrities (batch processing during fan-out on write).

Deep Dive 6: Content Delivery and Media Pipeline

Media upload flow:

1. Client requests a pre-signed URL from the API: POST /api/media/upload-url
2. API generates a pre-signed S3 URL (valid for 15 minutes)
3. Client uploads directly to S3 (bypasses our servers entirely)
4. S3 triggers a Lambda function (via S3 event notification)
5. Lambda runs the media pipeline: a. Content moderation (AWS Rekognition or custom ML model for NSFW detection) b. Image resizing: generate thumbnails (150x150, 600x600, 1200x1200) c. Video transcoding: multiple resolutions (480p, 720p, 1080p) via AWS MediaConvert d. Store all variants in S3 e. Invalidate CDN cache for the media URL
6. Lambda publishes MediaProcessed event
7. Post Service updates the post record with final media URLs

Serving: All media is served via CloudFront CDN. Origin is S3. Cache hit rate for popular content is > 95%.

---

What is Expected at Each Level

Mid-Level

• Design basic post creation and retrieval with a relational database
• Understand the concept of a feed (fetch posts from followed users)
• Identify that fetching from all followed users at read time is slow
• Propose some form of caching for the feed

Senior

• Explain fan-out on write vs fan-out on read with clear trade-offs
• Propose the hybrid approach for celebrities
• Design the feed cache in Redis with pagination
• Back-of-envelope calculations for fan-out write volume and Redis sizing
• Discuss feed ranking (at least a scoring formula)
• Handle the follow/unfollow flow including feed backfill/cleanup
• Media upload via pre-signed URLs and CDN

Staff+

• Design the ML-based ranking pipeline with latency constraints
• Hot-spot detection for viral content (Count-Min Sketch, adaptive caching)
• Social graph storage strategy at scale (dual-table sharding + Redis hybrid)
• Fan-out optimization: batch writes, pipelining, prioritization (active users first, dormant users lazy)
• Multi-region feed consistency: what happens if a user in Europe follows someone in Asia?
• Operational: monitoring feed freshness (time from post creation to appearance in follower feeds), alerting on fan-out lag, capacity planning for Redis cluster
• Content moderation pipeline and its impact on post visibility latency