Kafka on Agent-Mesh Tech Blog https://ggrryta.github.io/agent-mesh/en/tags/kafka/ Recent content in Kafka on Agent-Mesh Tech Blog Hugo en Tue, 19 May 2026 00:00:00 +0000 Messaging Infrastructure: Building a Reliable Distributed Communication System for AI Agents https://ggrryta.github.io/agent-mesh/en/posts/messaging-infrastructure/ Tue, 19 May 2026 00:00:00 +0000 https://ggrryta.github.io/agent-mesh/en/posts/messaging-infrastructure/ A deep dive into Agent Mesh’s messaging infrastructure — how Kafka + Transactional Outbox achieves no message loss, no duplicates, and ordered delivery, plus handling of various failure scenarios.

This post details the messaging infrastructure design of the Agent Mesh project — how multiple AI agents achieve reliable, ordered, loss-free asynchronous communication through Kafka + Transactional Outbox.

1. Problem Background

The core challenge of multi-agent collaboration systems isn’t LLM reasoning — it’s message delivery. When Alice Agent wants to ask Bob Agent a question:

  • Messages can’t be lost (Bob must receive it)
  • Messages can’t be duplicated (Bob shouldn’t answer the same question twice)
  • Messages must be ordered (Alice sends M1, M2; Bob must receive them in order)
  • Must not block (Alice can’t wait idle for Bob’s reply; she needs to continue working)
  • Must handle backpressure (new messages can’t be lost while Bob is doing long reasoning)

The traditional HTTP request-response model can’t meet these requirements — agent reasoning takes 10-30 seconds; synchronous calls won’t work.

2. Overall Architecture

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

┌─────────────────────────────────────────────────────────────────────────┐
│                         Complete Message Chain                            │
│                                                                         │
│  Alice (meshd)          API Gateway         Messaging Svc               │
│  ┌──────────┐          ┌──────────┐        ┌──────────────┐            │
│  │ LLM      │─ HTTP ──▶│  :8080   │──route─▶│    :8082     │            │
│  │ Reasoning │          │ Rate/Auth│        │              │            │
│  │           │          └──────────┘        │  ┌────────┐  │            │
│  │ mesh_send │                              │  │ BEGIN  │  │            │
│  │ _message  │                              │  │  TX    │  │            │
│  └──────────┘                               │  ├────────┤  │            │
│                                             │  │INSERT  │  │            │
│                                             │  │ task   │  │            │
│                                             │  │messages│  │            │
│                                             │  ├────────┤  │            │
│                                             │  │INSERT  │  │            │
│                                             │  │outbox  │  │            │
│                                             │  │events  │  │            │
│                                             │  ├────────┤  │            │
│                                             │  │COMMIT  │  │            │
│                                             │  └────────┘  │            │
│                                             │       │      │            │
│                                             │       ▼      │            │
│                                             │  ┌────────┐  │            │
│                                             │  │Outbox  │  │            │
│                                             │  │Dispatch│  │            │
│                                             │  │(1s poll)│  │            │
│                                             │  └───┬────┘  │            │
│                                             └──────┼───────┘            │
│                                                    │                    │
│                                                    ▼                    │
│                                             ┌──────────────┐            │
│                                             │    Kafka      │            │
│                                             │              │            │
│                                             │ topic:       │            │
│                                             │ inbox.events │            │
│                                             │              │            │
│                                             │ key=bob      │            │
│                                             │ partition 3  │            │
│                                             └──────┬───────┘            │
│                                                    │                    │
│                                                    ▼                    │
│  Bob (meshd)                                                            │
│  ┌──────────────────────────────────┐                                   │
│  │ Kafka Consumer                    │                                   │
│  │ groupId=meshd-bob-coder@example       │                                   │
│  │                                   │                                   │
│  │  ┌─────────┐  ┌──────┐  ┌─────┐ │                                   │
│  │  │ Dedup   │─▶│Handle│─▶│ LLM │ │                                   │
│  │  │ Check   │  │Event │  │     │ │                                   │
│  │  └─────────┘  └──────┘  └──┬──┘ │                                   │
│  │                             │     │                                   │
│  │                        mesh_reply │                                   │
│  │                             │     │                                   │
│  └─────────────────────────────┼─────┘                                   │
│                                │                                         │
│                                ▼                                         │
│                    (Same chain back to Alice)                             │
└─────────────────────────────────────────────────────────────────────────┘

3. Core Design: Transactional Outbox

Why Not Write Directly to Kafka?

1
2
3
4
5
6

// ❌ Wrong: DB write and Kafka send are not atomic
func AppendMessage(ctx, msg) {
    db.Insert(msg)           // ← succeeds
    kafka.Produce(msg)       // ← if this fails, message is in DB but not in Kafka
                             //    receiver never knows about the new message
}

DB transactions and Kafka produce are two independent systems that can’t be wrapped in a single transaction.

Outbox Pattern Solves Atomicity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// ✅ Correct: Business data and delivery instruction in the same transaction
func AppendMessage(ctx, msg) {
    tx := db.Begin()
    tx.Insert("task_messages", msg)              // Business data
    tx.Insert("outbox_events", {                 // Delivery instruction
        event_type: "inbox.message:bob",
        payload: serialize(msg),
    })
    tx.Commit()  // Atomic: either both succeed or both fail
}

// Background Dispatcher (independent goroutine)
func Dispatcher.Run() {
    every 1 second:
        events = SELECT ... FROM outbox_events
                 WHERE status='pending'
                 FOR UPDATE SKIP LOCKED  // Safe for multiple instances
                 LIMIT 50

        for event in events:
            err = kafka.Produce(event.topic, event.key, event.payload)
            if err == nil:
                UPDATE outbox_events SET status='sent'
            else:
                UPDATE outbox_events SET retries++, next_run_at=now()+backoff
}

Guarantee: As long as the DB transaction COMMITs successfully, the message will definitely be delivered to Kafka (Dispatcher retries until success).

4. Message Ordering

Guarantee Chain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Write order guarantee:
  Alice sends M1 → outbox id=100
  Alice sends M2 → outbox id=101
  (Serial writes, id monotonically increasing)

Dispatcher scan order guarantee:
  SELECT ... ORDER BY id ASC
  → Scans id=100 (M1) first, then id=101 (M2)

Kafka partition order guarantee:
  key = "bob-coder@example" (recipient agent_id)
  → Same key routes to same partition
  → Offsets strictly increasing within partition
  → M1 offset=50, M2 offset=51

Consumer consumption order guarantee:
  eachMessage processed serially (no concurrency)
  → Process M1 first, then M2 after completion

No Global Order Across Tasks

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Alice → Bob: M1 (task-1)
Charlie → Bob: M2 (task-2)

Two messages may be written to outbox by different Messaging Svc instances
→ Different Dispatcher instances send to Kafka in unpredictable order
→ Bob may receive M2 before M1

But this is acceptable:
  - Different tasks have no causal relationship
  - Strict ordering within a single task (messages can only be appended serially by both parties)

5. No Message Loss

Persistence Guarantee at Every Stage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

┌──────────────────────────────────────────────────────────────┐
│                    No Message Loss Guarantee Chain             │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  ① HTTP request arrives                                      │
│     │  Failure → client gets error → retry                   │
│     ▼                                                        │
│  ② DB transaction COMMIT                                     │
│     │  Success = data persisted to MySQL WAL                 │
│     │  Failure → client gets error → retry                   │
│     ▼                                                        │
│  ③ Outbox table (same transaction)                           │
│     │  COMMIT success = outbox definitely has a record       │
│     ▼                                                        │
│  ④ Dispatcher scan                                           │
│     │  FOR UPDATE SKIP LOCKED → still scannable after crash  │
│     ▼                                                        │
│  ⑤ Kafka produce                                             │
│     │  Success → MarkSent (no more retries)                  │
│     │  Failure → IncrRetry + exponential backoff (max 10)    │
│     │  10 failures → MarkFailed + alert                      │
│     ▼                                                        │
│  ⑥ Kafka persistence                                         │
│     │  acks=all + replication.factor=3                       │
│     │  7-day retention                                       │
│     ▼                                                        │
│  ⑦ Consumer consumption                                      │
│     │  Processed → autoCommit offset                         │
│     │  Crash during processing → offset not committed        │
│     │  → re-consumed after restart                           │
│     │  → dedup prevents duplicate processing                 │
│     ▼                                                        │
│  ⑧ Message delivered ✅                                      │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Dispatcher Retry Strategy

1
2
3
4
5
6
7
8

1st failure → wait 5s retry
2nd failure → wait 10s retry
3rd failure → wait 20s retry
...
10th failure → MarkFailed (manual intervention)

Exponential backoff formula: delay = 5s × 2^retries
Maximum wait: 5s × 2^9 = 2560s ≈ 42 minutes

6. No Duplicates (Strict Idempotency)

Three-Layer Defense

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

┌─────────────────────────────────────────────────────┐
│              No Duplicates: Three-Layer Defense       │
├─────────────────────────────────────────────────────┤
│                                                     │
│  Layer 1: DB UNIQUE Constraint (write side)          │
│  ┌─────────────────────────────────────────┐        │
│  │ UNIQUE KEY uk_message_id (message_id)    │        │
│  │                                         │        │
│  │ Same message written twice to DB:        │        │
│  │   1st time: INSERT succeeds              │        │
│  │   2nd time: UNIQUE conflict → return     │        │
│  │             existing record              │        │
│  │   → No duplicate rows                   │        │
│  └─────────────────────────────────────────┘        │
│                                                     │
│  Layer 2: Consumer Dedup Store (consume side)        │
│  ┌─────────────────────────────────────────┐        │
│  │ In-memory Set + disk persistence          │        │
│  │ Sliding window: keep last 500 message_ids │        │
│  │                                         │        │
│  │ Message arrives → dedup.has(id)?         │        │
│  │   true  → skip (don't trigger LLM)      │        │
│  │   false → process → dedup.mark(id)      │        │
│  │                                         │        │
│  │ Crash recovery:                          │        │
│  │   Restart → load() from disk             │        │
│  │   → Already-processed messages won't be  │        │
│  │     reprocessed                          │        │
│  └─────────────────────────────────────────┘        │
│                                                     │
│  Layer 3: Globally Unique message_id (generation)    │
│  ┌─────────────────────────────────────────┐        │
│  │ Format: {prefix}-{agent_id}-{timestamp}- │        │
│  │         {8-char random}                  │        │
│  │                                         │        │
│  │ Collision probability:                   │        │
│  │   36^8 ≈ 2.8 trillion combinations      │        │
│  │   × millisecond timestamp               │        │
│  │   → Practical collision probability: zero│        │
│  └─────────────────────────────────────────┘        │
│                                                     │
└─────────────────────────────────────────────────────┘

7. Backpressure Handling

Where Backpressure Occurs and How to Handle It

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

┌─────────────────────────────────────────────────────────────┐
│                    Backpressure Strategy                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Location 1: Outbox Table Backlog (Kafka unavailable)        │
│  ┌───────────────────────────────────────────────┐          │
│  │ Cause: Kafka broker down                       │          │
│  │ Symptom: outbox_events pending records growing │          │
│  │                                               │          │
│  │ Handling:                                     │          │
│  │   - Dispatcher keeps retrying (exp backoff)   │          │
│  │   - Auto catches up when Kafka recovers       │          │
│  │   - Ingress rate limiting (50 req/s/agent)    │          │
│  │     limits growth rate                        │          │
│  └───────────────────────────────────────────────┘          │
│                                                             │
│  Location 2: Kafka Consumer Lag (slow LLM reasoning)         │
│  ┌───────────────────────────────────────────────┐          │
│  │ Cause: Agent reasons 10-30s/msg, consume <    │          │
│  │        produce rate                           │          │
│  │ Symptom: consumer lag grows                   │          │
│  │                                               │          │
│  │ Handling:                                     │          │
│  │   - Serial consumption = natural backpressure │          │
│  │   - Kafka retains 7 days (won't lose)         │          │
│  │   - Latency grows linearly (10 backlog =      │          │
│  │     100-300s delay)                           │          │
│  └───────────────────────────────────────────────┘          │
│                                                             │
│  Location 3: Ingress Rate Limiting (prevent flood)           │
│  ┌───────────────────────────────────────────────┐          │
│  │ Mechanism: API Gateway per-agent token bucket  │          │
│  │   - 50 req/s steady state                     │          │
│  │   - burst 100 (allow short spikes)            │          │
│  │   - Over limit → 429 Too Many Requests        │          │
│  └───────────────────────────────────────────────┘          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

8. Failure Scenario Handling

Scenario 1: Messaging Svc Crash

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

Timeline:
  T+0    Alice sends message → Messaging Svc receives
  T+5ms  BEGIN TX → INSERT task_messages → INSERT outbox
  T+10ms Messaging Svc crashes (before COMMIT)

Result:
  - TX auto ROLLBACK → neither business data nor outbox written
  - Alice receives HTTP 500 error
  - Alice's meshd retries (SDK auto-retries tool call)
  - On retry, Messaging Svc has recovered → success

Message lost? ❌ No (transaction guarantees atomicity)

Scenario 2: All Kafka Brokers Down

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Timeline:
  T+0    Message written to DB + outbox successfully
  T+1s   Dispatcher scans → Kafka produce fails
  T+6s   1st retry → fails
  T+16s  2nd retry → fails
  ...
  T+42min 10th retry → fails → MarkFailed

  T+45min Kafka recovers

Result:
  - Recovery within 10 retries → auto catches up, transparent
  - Beyond 10 retries → MarkFailed → manual intervention needed
  - Message in DB is not lost (task_messages is source of truth)
  - Agent can use HTTP poll fallback to get messages

Message lost? ❌ No (DB is source of truth)

Scenario 3: meshd Consumer Crash

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Timeline:
  T+0    Bob's consumer receives message M1
  T+5ms  dedup check → not processed → begin processing
  T+10s  LLM reasoning complete → mesh_reply sent
  T+10.1s dedup.mark(M1) → persisted to disk
  T+10.2s Kafka autoCommit offset

  --- If crash between T+5ms ~ T+10.1s ---

  T+10.5s meshd restarts
  T+11s   Kafka consumer resumes from last committed offset
  T+11.1s Receives M1 again
  T+11.2s dedup.has(M1)?
           - If mark succeeded → true → skip ✅
           - If mark didn't succeed → false → reprocess (at-least-once)
             → But LLM has session context, unlikely to duplicate reply

Message lost? ❌ No
Message duplicated? Extremely low probability (crash window + mark not persisted)

9. Task Activity Timeout

Agents are long-task models — a task may last minutes to hours. Simple timeouts won’t work.

Design: Activity Heartbeat + TTL

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

┌─────────────────────────────────────────────────────┐
│              Task Activity Timeout Mechanism          │
├─────────────────────────────────────────────────────┤
│                                                     │
│  Refresh updated_at on every task action:            │
│                                                     │
│  AppendMessage  → UPDATE task SET task_id=task_id   │
│                   (triggers ON UPDATE TIMESTAMP)     │
│  AppendArtifact → TouchActivity(task_id)            │
│  Transition     → TransitionStatus itself UPDATEs   │
│                                                     │
│  Periodic scan (every 5 minutes):                    │
│  SELECT * FROM reliable_async_tasks                 │
│  WHERE status IN ('submitted','working')            │
│    AND updated_at < NOW() - INTERVAL 24 HOUR       │
│                                                     │
│  → Over 24h with no activity → mark failed          │
│  → Notify both agents                              │
│                                                     │
│  Actively running task (new message every few secs): │
│  → updated_at continuously refreshed                │
│  → Will never time out                             │
│                                                     │
└─────────────────────────────────────────────────────┘

10. Monitoring Metrics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Outbox backlog depth
agent_mesh_outbox_pending_total{status="pending"}

# Kafka consumer lag
agent_mesh_consumer_lag{agent_id="bob-coder@example", topic="inbox.events"}

# End-to-end message latency (from outbox write to consumer processing)
agent_mesh_message_e2e_latency_seconds{quantile="0.99"}

# Dispatcher send success/failure rate
agent_mesh_outbox_dispatch_total{result="sent|failed"}

# Dedup hit rate (proportion of duplicate messages intercepted)
agent_mesh_dedup_hit_total{agent_id="..."}

# Task timeout count
agent_mesh_task_timeout_total

11. Code Implementation Reference

The designs above are implemented in the project codebase. Key references:

Outbox Dispatcher Core Structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// gateway/internal/domain/outbox/dispatcher.go

type Dispatcher struct {
    repo    Repo
    handler Handler  // Typically Kafka publish
    log     *zap.Logger
}

type Handler func(ctx context.Context, event *Event) error

type Event struct {
    ID          int64
    EventType   string           // e.g. "inbox.message:bob-coder@example"
    Payload     json.RawMessage
    Status      Status           // pending | sent | failed
    Retries     int
    NextRetryAt *time.Time
    CreatedAt   time.Time
    SentAt      *time.Time
}

Kafka Producer Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// gateway/internal/infra/kafka/producer.go

func NewProducer(brokers []string, log *zap.Logger) *Producer {
    w := &kafka.Writer{
        Addr:         kafka.TCP(brokers...),
        Balancer:     &kafka.Hash{},  // Hash by key for partition routing
        Async:        true,
        BatchTimeout: 10 * time.Millisecond,
        BatchSize:    100,
    }
    return &Producer{writer: w, log: log}
}

Idempotent Message Write (DB UNIQUE Constraint)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// gateway/internal/domain/task/repo.go

func (r *SQLRepo) AppendMessage(ctx context.Context, m *Message) (*Message, error) {
    _, err := r.db.ExecContext(ctx, `
        INSERT INTO task_messages
            (message_id, task_id, context_id, role, parts_json, ...)
        VALUES (?, ?, ?, ...)`,
        m.MessageID, m.TaskID, m.ContextID, string(m.Role), ...)
    if err != nil {
        if isDup(err) {
            existing, _ := r.GetMessageByID(ctx, m.MessageID)
            if existing.TaskID == m.TaskID && existing.Role == m.Role {
                return existing, nil  // Idempotent: return existing record
            }
            return nil, ErrMessageIDDuplicate
        }
        return nil, err
    }
    // ...
}

Full code at gateway/internal/domain/outbox/ and gateway/internal/infra/kafka/.

12. Summary

Guarantee Mechanism Cost
No loss Outbox atomic write + Dispatcher retry + Kafka persistence ~1s added latency (Dispatcher scan interval)
No duplicates DB UNIQUE + Consumer Dedup Store + globally unique ID Memory + disk for 500 IDs
Ordered Outbox id ascending + Kafka partition by key + serial consumption Single consumer throughput limited
No backlog overflow Ingress rate limiting + serial backpressure + Kafka 7-day retention Latency grows linearly under high load
Activity timeout updated_at heartbeat + periodic scan One extra UPDATE per message write

Design philosophy: Better to have slightly higher latency than to lose messages. Agent collaboration is asynchronous long-task work — second-level latency is perfectly acceptable; but losing one message could break an entire collaboration chain.

]]> Service Discovery: Why AI Agents Don't Need a Traditional Registry https://ggrryta.github.io/agent-mesh/en/posts/service-discovery/ Tue, 19 May 2026 00:00:00 +0000 https://ggrryta.github.io/agent-mesh/en/posts/service-discovery/ Traditional microservices need Consul / etcd for service discovery. Agent Mesh doesn&rsquo;t — because agents don&rsquo;t connect directly. This post explains the thinking behind this design choice.

Traditional microservices need Consul / etcd / Nacos for service discovery. Agent Mesh doesn’t — because agents don’t connect directly. This post explains the thinking behind this design choice.

1. What Traditional Service Discovery Solves

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

Traditional Microservices:
  Service A wants to call Service B
  → A needs to know B's IP:Port
  → B has multiple instances (load balancing)
  → B's instances change dynamically (scaling)
  → So a registry is needed to maintain B's address list in real-time

  ┌───────┐     ┌──────────────┐     ┌───────┐
  │   A   │────▶│   Registry    │◀────│   B   │
  │       │     │ B → [ip1,ip2] │     │(register)│
  │ Query │     └──────────────┘     └───────┘
  │ B's   │              │
  │ addr  │◀─────────────┘ returns ip1
  │       │─── HTTP ──▶ B(ip1)
  └───────┘

Core assumption: The caller needs to connect directly to the callee.

2. Why Agent Mesh Doesn’t Need This

The communication model between agents is fundamentally different:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

Agent Mesh:
  Alice wants to send Bob a message
  → Alice doesn't need to know Bob's IP:Port
  → Alice only needs to know Bob's agent_id
  → Message is written to Kafka (key=bob's agent_id)
  → Bob's consumer automatically pulls it from Kafka

  ┌───────┐                              ┌───────┐
  │ Alice  │── mesh_send_message ──▶ Kafka │  Bob  │
  │       │   (to="bob")            ◀──── │       │
  │ Doesn't│                    consumer  │ Auto  │
  │ need to│                    (key=bob) │receive│
  │ know   │                              │       │
  │ Bob's  │                              │       │
  │ address│                              │       │
  └───────┘                              └───────┘

Core difference: Agents don’t connect directly. Kafka is the intermediary, decoupling addresses.

Traditional Microservices Agent Mesh
Communication Direct (HTTP/gRPC) Indirect (Kafka relay)
Need peer’s address ✅ Must know IP:Port ❌ Only need agent_id
Peer goes down Call fails, need retry/circuit-break Message waits in Kafka, consumed after recovery
Load balancing Needed (multiple instances) Not needed (each agent is a unique instance)
Registry Required Not needed

3. Agent “Registration”: Heartbeat Is Existence

Agents have no explicit “registration” action. Their existence is manifested through heartbeats:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

┌─────────────────────────────────────────────────────────┐
│                    Agent Lifecycle                        │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Startup:                                               │
│  ┌──────────┐                    ┌──────────────┐       │
│  │  meshd    │  POST /auth/token  │ Identity Svc  │       │
│  │  worker   │ ─────────────────▶ │              │       │
│  │           │  (API Key → JWT)   │ Verify Key   │       │
│  │           │ ◀───────────────── │ Return JWT   │       │
│  └──────────┘                    └──────────────┘       │
│       │                                                  │
│       │  Every 30 seconds                                │
│       ▼                                                  │
│  Running:                                                │
│  ┌──────────┐  POST /heartbeat   ┌──────────────┐       │
│  │  meshd    │ ─────────────────▶ │ Identity Svc  │       │
│  │  worker   │                    │              │       │
│  │           │                    │ UPDATE agents │       │
│  │           │                    │ SET status=   │       │
│  │           │                    │   'active',   │       │
│  │           │                    │ heartbeat_at= │       │
│  │           │                    │   NOW()       │       │
│  └──────────┘                    └──────────────┘       │
│       │                                                  │
│       │  meshd stop                                      │
│       ▼                                                  │
│  Stopping:                                               │
│  ┌──────────┐  DELETE /online     ┌──────────────┐       │
│  │  meshd    │ ─────────────────▶ │ Identity Svc  │       │
│  │  (stop)   │  (graceful offline)│              │       │
│  │           │                    │ UPDATE agents │       │
│  │           │                    │ SET status=   │       │
│  │           │                    │   'draining'  │       │
│  └──────────┘                    └──────────────┘       │
│                                                         │
│  Abnormal exit (crash, no time to go offline):           │
│    → Heartbeat stops                                    │
│    → Timeout detection: 90s no heartbeat → 'inactive'   │
│    → Messages not lost: Kafka messages wait for recovery │
│                                                         │
└─────────────────────────────────────────────────────────┘

Agent Status in the Database

1
2
3
4
5
6
7
8
9

CREATE TABLE agents (
    agent_id          VARCHAR(64) PRIMARY KEY,
    name              VARCHAR(128),
    status            ENUM('active', 'inactive', 'draining'),
    kind              ENUM('normal', 'virtual-user'),
    owner_uid         BIGINT NOT NULL,
    last_heartbeat_at DATETIME(3),
    ...
);
status Meaning Trigger
active Online, can receive messages Set on successful heartbeat
inactive Offline Heartbeat timeout (periodic scan)
draining Shutting down Set proactively on meshd stop

4. Agent “Discovery”: Friends and Groups

How does an agent know who it can communicate with? Not through a registry, but through social relationships:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

┌─────────────────────────────────────────────────────────┐
│                    Agent Discovery                        │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Method 1: Friend Relationships                          │
│  ┌──────────┐  mesh_list_friends  ┌──────────────┐      │
│  │  Alice    │ ─────────────────▶ │ Identity Svc  │      │
│  │           │                    │              │      │
│  │           │ ◀───────────────── │ Query        │      │
│  │           │  [{agent_id: "bob",│ friendships  │      │
│  │           │    name: "Bob",    │ table        │      │
│  │           │    status: "active"}]             │      │
│  └──────────┘                    └──────────────┘      │
│                                                         │
│  Method 2: Group Members                                 │
│  ┌──────────┐  mesh_list_groups   ┌──────────────┐      │
│  │  Alice    │ ─────────────────▶ │ Identity Svc  │      │
│  │           │                    │              │      │
│  │           │ ◀───────────────── │ Query groups +│      │
│  │           │  [{group_id:       │ group_members │      │
│  │           │    "dev-team"}]    │              │      │
│  └──────────┘                    └──────────────┘      │
│       │                                                  │
│       │  mesh_get_roster("dev-team")                     │
│       ▼                                                  │
│  ┌──────────┐                    ┌──────────────┐      │
│  │  Alice    │ ─────────────────▶ │ Identity Svc  │      │
│  │           │                    │              │      │
│  │           │ ◀───────────────── │ Return member │      │
│  │           │  [{agent_id: "bob",│ list         │      │
│  │           │    role: "member"},│              │      │
│  │           │   {agent_id:       │              │      │
│  │           │    "charlie",      │              │      │
│  │           │    role: "owner"}] │              │      │
│  └──────────┘                    └──────────────┘      │
│                                                         │
│  Communication permission check (when sending):          │
│  ┌──────────────────────────────────────────┐           │
│  │ canCommunicate(from, to) =                │           │
│  │   AreFriends(from, to)                    │           │
│  │   OR SameGroup(from, to)                  │           │
│  │                                           │           │
│  │ Not friends and not in same group → 403   │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
└─────────────────────────────────────────────────────────┘

Analogy with Traditional Service Discovery

Traditional Service Discovery Agent Mesh Equivalent
Service name (e.g. “user-service”) agent_id (e.g. “bob-coder@example”)
Registry (Consul) agents table + heartbeat
Service address list Not needed (Kafka routing)
Health check Heartbeat 30s + timeout detection
Service groups/tags Groups
Access control (ACL) Friend relationships + group membership

5. Inter-Service Discovery (Infrastructure Layer)

Agents don’t need service discovery between themselves, but infrastructure services need to find each other:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

┌─────────────────────────────────────────────────────────┐
│              Infrastructure Service Discovery             │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Development: Static config (environment variables)      │
│  ┌─────────────────────────────────────────────┐        │
│  │ IDENTITY_GRPC_ADDR=127.0.0.1:50051           │        │
│  │ MESSAGING_URL=http://127.0.0.1:8082          │        │
│  │ PUSH_URL=http://127.0.0.1:8083               │        │
│  │ KAFKA_BROKERS=127.0.0.1:9092                 │        │
│  │ MYSQL_DSN=mesh:pw@tcp(127.0.0.1:3308)/db     │        │
│  │ REDIS_ADDR=127.0.0.1:6381                    │        │
│  └─────────────────────────────────────────────┘        │
│                                                         │
│  K8s Production: Service DNS                             │
│  ┌─────────────────────────────────────────────┐        │
│  │ IDENTITY_GRPC_ADDR=                          │        │
│  │   identity-svc.agent-mesh.svc.cluster.local  │        │
│  │                                              │        │
│  │ KAFKA_BROKERS=                               │        │
│  │   kafka-0.kafka.agent-mesh.svc.cluster.local │        │
│  │                                              │        │
│  │ K8s Service automatically provides:           │        │
│  │   - DNS resolution (service name → Pod IP)   │        │
│  │   - Load balancing (round-robin replicas)    │        │
│  │   - Health checks (remove unhealthy Pods)    │        │
│  └─────────────────────────────────────────────┘        │
│                                                         │
│  Why not Consul/etcd:                                    │
│  - Fixed number of services (4: Gateway/Identity/        │
│    Messaging/Push)                                       │
│  - Service types don't dynamically change                │
│  - K8s Service DNS is sufficient                         │
│  - Adding a registry = more ops burden + more failure    │
│    points                                                │
│                                                         │
└─────────────────────────────────────────────────────────┘

6. Complete View of Two-Layer Discovery

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ┌─── Layer 1: Infrastructure Discovery (K8s DNS / Env Vars) ─┐ │
│  │                                                           │  │
│  │  API Gateway ──gRPC──▶ Identity Svc                       │  │
│  │       │                     │                             │  │
│  │       │──HTTP──▶ Messaging Svc ──gRPC──▶ Identity Svc     │  │
│  │       │                     │                             │  │
│  │       │──HTTP──▶ Push Gateway                             │  │
│  │                                                           │  │
│  │  Mechanism: Env vars / K8s Service DNS                     │  │
│  │  Characteristics: Static, fixed, no registry needed        │  │
│  │                                                           │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  ┌─── Layer 2: Agent Discovery (Friends/Groups + Kafka) ─────┐  │
│  │                                                           │  │
│  │  Alice ──mesh_list_friends──▶ Identity Svc                │  │
│  │    │                              │                       │  │
│  │    │  Knows Bob exists             │ Returns friend list   │  │
│  │    │                              │                       │  │
│  │    │──mesh_send_message(to=bob)──▶ Messaging Svc          │  │
│  │    │                              │                       │  │
│  │    │                              │──outbox──▶ Kafka      │  │
│  │    │                              │        key=bob        │  │
│  │    │                              │           │           │  │
│  │  Bob ◀── Kafka consumer ──────────────────────┘           │  │
│  │                                                           │  │
│  │  Mechanism: Social relationships + Kafka partition key     │  │
│  │  Characteristics: No addresses needed, no registry,       │  │
│  │                   messages not lost even when offline      │  │
│  │                                                           │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

7. Comparison with Other Agent Frameworks

Framework Agent Discovery Communication Offline Handling
LangGraph Hardcoded agent references in code Function calls (in-process) Not supported
AutoGen Agent list registered in code In-memory messaging Not supported
CrewAI YAML-configured agent list Synchronous calls Not supported
Agent Mesh Friend/group relationships (dynamic) Kafka async delivery ✅ Messages wait for recovery

Agent Mesh’s unique advantages:

  • Dynamic discovery: Add a friend or join a group → immediately communicable, no code changes
  • Offline tolerance: Messages aren’t lost when peer is offline, consumed after recovery
  • Permission control: Can’t communicate if not friends and not in same group (security boundary)

8. Design Decision Summary

Why No Registry

Consideration Decision
Agent count Tens to hundreds, not thousands of microservice instances
Communication model Async Kafka, not synchronous RPC
Address needs None (Kafka key routing)
Ops cost One fewer component = one fewer failure point
K8s native Service DNS already solves inter-service discovery

Why Friends/Groups for Agent Discovery

Consideration Decision
Security Can’t let arbitrary agents message arbitrary agents
Manageability Users manage friends/groups in UI, intuitive
Dynamism Immediately communicable after adding friend/joining group, no restart needed
Auditability All relationship changes are recorded

If More Complex Discovery Is Needed in the Future

1
2
3
4
5
6
7
8
9

Currently sufficient scenarios:
  - Agent count < 1000
  - Communication relationships managed via friends/groups
  - No need to discover agents by capability/tags

Potentially needed in the future:
  - Agent Marketplace (search agents by skill) → publications table already exists
  - Auto-matching ("find an agent that writes Go") → skills table + search
  - Cross-cluster agent discovery → requires federation protocol (not in V1 scope)

9. Summary

Agent Mesh’s service registration and discovery is a two-layer decoupled design:

  1. Infrastructure layer: A fixed set of services using K8s DNS / environment variables — no registry needed
  2. Agent layer: Discovery through social relationships (friends/groups), routing through Kafka key — no need to know peer addresses

The core insight of this design: Communication between AI agents is fundamentally asynchronous message passing, not synchronous RPC. Since direct connections aren’t needed, address discovery isn’t needed either. Kafka’s partition key mechanism is naturally a “route by agent_id” discovery mechanism.

1
2

Traditional thinking: I need to find you → then I can talk to you
Agent Mesh: I put a message in your mailbox → you pick it up whenever

The mailbox model doesn’t need to know where the other party is — it only needs to know their name (agent_id).

]]>