Netdata Zulip Bot#

A production-ready webhook service that receives notifications from Netdata Cloud and forwards them to Zulip channels. Designed to run behind a reverse proxy (like Caddy) that handles HTTPS and mutual TLS authentication.

Features#

• 🔗 Reverse Proxy Ready: HTTP service designed to run behind Caddy/nginx
• 🤝 Mutual TLS Support: When configured with reverse proxy
• 📊 Rich Formatting: Beautiful Zulip messages with emojis and markdown
• 🏷️ Topic Organization: Automatic topic routing by severity level
• 📝 Structured Logging: JSON-structured logs for monitoring
• ⚡ High Performance: FastAPI-based webhook endpoint
• 🔧 Flexible Configuration: Support for .zuliprc files or environment variables
• ✅ Webhook Verification: Built-in Netdata challenge/response handling

Quick Start#

1. Install Dependencies#

# Using uv (recommended)
uv sync

2. Create Configuration#

# Generate sample configuration files
uv run netdata-zulip-bot --create-config

# Copy and customize
cp .zuliprc.sample ~/.zuliprc
cp .env.sample .env

3. Configure Zulip Settings#

Edit ~/.zuliprc:

[api]
site=https://yourorg.zulipchat.com
email=netdata-bot@yourorg.zulipchat.com  
key=your-zulip-api-key
stream=netdata-alerts

4. Configure Environment Variables#

Edit .env file or set environment variables:

# Server configuration (HTTP only)
export SERVER_HOST=0.0.0.0
export SERVER_PORT=8080

# Required: Netdata webhook challenge secret
export SERVER_CHALLENGE_SECRET=your-challenge-secret-here

# Optional: Override Zulip stream
export ZULIP_STREAM=netdata-alerts

5. Run the Service#

# Start the HTTP service
uv run netdata-zulip-bot

# Or with custom configuration
uv run netdata-zulip-bot --zuliprc /path/to/.zuliprc

# The service runs on HTTP (default: localhost:8080)
# Use a reverse proxy like Caddy for HTTPS and mutual TLS

Configuration#

Zulip Configuration#

The bot supports two configuration methods:

Method 1: Zuliprc File (Recommended)#

Create ~/.zuliprc:

[api]
site=https://yourorg.zulipchat.com
email=netdata-bot@yourorg.zulipchat.com
key=your-zulip-api-key
stream=netdata-alerts

Method 2: Environment Variables#

export ZULIP_SITE=https://yourorg.zulipchat.com
export ZULIP_EMAIL=netdata-bot@yourorg.zulipchat.com
export ZULIP_API_KEY=your-api-key
export ZULIP_STREAM=netdata-alerts

Use the --env-config flag to use environment variables instead of zuliprc:

uv run netdata-zulip-bot --env-config

Server Configuration#

Set these environment variables:

• SERVER_HOST: Bind address (default: 0.0.0.0)
• SERVER_PORT: HTTP port (default: 8080)
• SERVER_CHALLENGE_SECRET: Netdata webhook challenge secret (required)

Reverse Proxy Setup#

The bot is designed to run behind a reverse proxy that handles HTTPS and mutual TLS:

Using Caddy (Recommended)#

1. Update Caddyfile with your domain name
2. Place Netdata CA certificate in netdata-ca.pem
3. Run both services:

# Start the bot
uv run netdata-zulip-bot &

# Start Caddy
caddy run --config Caddyfile

Using Docker Compose#

docker-compose up -d

Message Format#

Alert Notifications#

Messages are posted to topics based on severity level:

• Topic: critical, warning, or clear
• Format: Rich markdown with alert details, timestamps, and links

Example:

🔴 **High CPU Usage**

**Space:** production
**Chart:** system.cpu
**Context:** cpu utilization
**Severity:** Critical
**Time:** 2024-01-15 14:30:00 UTC

**Details:** CPU usage has exceeded 90% for 5 minutes

**Summary:** Critical alert: High CPU usage detected

[View Alert](https://app.netdata.cloud/spaces/...)

Reachability Notifications#

Messages are posted to the reachability topic:

❌ **Host Unreachable**

**Host:** web-server-01
**Status:** ❌ Unreachable  
**Severity:** Critical

**Summary:** Host web-server-01 is no longer reachable

[View Host](https://app.netdata.cloud/...)

Deployment#

Systemd Service#

See examples/netdata-zulip-bot.service for a complete systemd service configuration.

Automated Setup#

Use the provided setup script:

sudo ./scripts/setup.sh --domain your-domain.com --email admin@example.com

Docker#

The included Dockerfile and docker-compose.yml provide a complete setup with Caddy reverse proxy:

docker-compose up -d

Security#

Architecture#

The bot uses a security-focused architecture:

1. HTTP Backend: Simple HTTP service with no direct internet exposure
2. Reverse Proxy: Caddy handles HTTPS, certificates, and client authentication
3. Mutual TLS: Client certificate validation at the reverse proxy level

Webhook Security#

• Challenge/Response: Built-in Netdata webhook verification using HMAC-SHA256
• Payload Validation: Strict payload parsing and validation
• Request Logging: Comprehensive logging of all webhook requests
• Error Handling: Secure error responses without information disclosure

SSL Certificate Management#

SSL certificates are managed by the reverse proxy (Caddy):

1. Automatic Provisioning: Caddy obtains Let's Encrypt certificates
2. Automatic Renewal: Built-in certificate renewal
3. Mutual TLS: Client certificate validation using Netdata CA certificate

Monitoring#

The service provides structured JSON logging for easy monitoring:

{
  "timestamp": "2024-01-15T14:30:00.000Z",
  "level": "info",
  "event": "Message sent to Zulip",
  "stream": "netdata-alerts",
  "topic": "critical",
  "message_id": 12345
}

Health Check#

# Direct HTTP check (backend service)
curl http://localhost:8080/health

# Through reverse proxy
curl https://your-domain.com/health

Response:

{
  "status": "healthy",
  "service": "netdata-zulip-bot"
}

Development#

Running Tests#

uv run python -m pytest tests/ -v

Code Formatting#

uv run black .
uv run ruff check .

Local Development#

For development, you can run the HTTP service directly:

# Set required environment variables
export SERVER_CHALLENGE_SECRET=test-secret

# Run the service
uv run netdata-zulip-bot

# Test webhook endpoint
curl -X POST http://localhost:8080/webhook/netdata?crc_token=test123

Troubleshooting#

Common Issues#

1. Configuration Issues

   • Ensure SERVER_CHALLENGE_SECRET is set (required for Netdata webhook verification)
   • Verify .zuliprc file contains all required fields
   • Check that Zulip bot has permission to post to the configured stream

2. Reverse Proxy Issues

   • Ensure Caddy configuration uses correct domain name
   • Verify Netdata CA certificate is properly configured
   • Check that port 80 is accessible for Let's Encrypt challenges

3. Webhook Not Receiving Data

   • Verify Netdata Cloud webhook URL points to your reverse proxy
   • Check webhook challenge secret matches configuration
   • Review service logs for error messages

Logs#

View service logs:

sudo journalctl -u netdata-zulip-bot -f

License#

MIT License - see LICENSE file for details.