# Architecture Design

System architecture and technology stack for CiteFlow.

## Overview

CiteFlow is a modern web application built with Next.js, designed to track AI bot activity and referral traffic. The architecture follows a **three-layer database design** for optimal performance at scale.

## Product Positioning

**CiteFlow = AI Bots + AI Referrals + AI Traffic Intelligence**

### Target Scale
- **Multi-site**: Support for thousands of tracked websites
- **Bot Visits**: Millions of AI bot visits per month
- **Referrals**: Hundreds of thousands of AI referrals
- **Pages**: Page-centric data model as the core entity

## Technology Stack

| Layer | Technology | Version |
|-------|------------|---------|
| Frontend | Next.js 16, React, Tailwind CSS | Next.js 16+ |
| Backend | Next.js API Routes | Next.js 16+ |
| Database | PostgreSQL with Prisma ORM | PostgreSQL 16+ |
| Auth | Clerk | Latest |
| Payments | Stripe | Latest |
| Deployment | Vercel | Latest |

## Three-Layer Database Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                        CiteFlow Platform                        │
├─────────────────────────────────────────────────────────────────┤
│  Frontend (Next.js App Router)                                  │
│  ├── Landing Page                                               │
│  ├── Dashboard (Overview, Pages, Bots, Referrals, Settings)    │
│  └── Documentation                                              │
├─────────────────────────────────────────────────────────────────┤
│  API Layer                                                      │
│  ├── /api/tracker - Event collection                            │
│  ├── /api/bots - Bot activity data (BotVisit)                  │
│  ├── /api/pages - Page discovery data                          │
│  ├── /api/referrals - AI referral data                          │
│  ├── /api/overview - Dashboard stats                            │
│  ├── /api/stripe - Payment handling                             │
│  └── /api/maintenance - Cleanup operations                     │
├─────────────────────────────────────────────────────────────────┤
│  Service Layer                                                  │
│  ├── AnalyticsService - Data aggregation                        │
│  ├── BotService - Bot tracking logic                            │
│  ├── SiteService - Site management                              │
│  └── ClassificationEngine - AI traffic classification          │
├─────────────────────────────────────────────────────────────────┤
│  Repository Layer                                               │
│  ├── SiteRepository - Site data access                          │
│  ├── BotVisitRepository - Analytics data access                 │
│  ├── CrawlerLogRepository - Raw log access                      │
│  └── PageRepository - Page data access                          │
├─────────────────────────────────────────────────────────────────┤
│  Database (PostgreSQL) - Three-Layer Architecture               │
│  ├─ RAW LAYER (CrawlerLog) - 30 day retention                  │
│  ├─ ANALYTICS LAYER (BotVisit, Referral) - Permanent           │
│  └─ AGGREGATION LAYER (Page, Session, Event) - Permanent       │
└─────────────────────────────────────────────────────────────────┘
```

## Database Layer Design

| Layer | Tables | Retention | Purpose |
|-------|--------|-----------|---------|
| **Raw** | CrawlerLog | 30 days | Debugging raw crawler requests |
| **Analytics** | BotVisit, Referral | Permanent | Business facts for reporting |
| **Aggregation** | Page, Session, Event | Permanent | Aggregated statistics |

**Important Rule:** Dashboard queries MUST use Analytics/Aggregation tables. Raw logs are for debugging only.

## Data Flow

### Tracker Flow

```
Website → tracker.js → /api/tracker → Session/Event/Referral tables
```

1. User visits website with tracker installed
2. Tracker detects AI source from referrer/header
3. Events sent to `/api/tracker`
4. Data stored in Analytics/Aggregation tables

### Bot Tracking Flow

```
Cloudflare Worker → /api/bots → CrawlerLog + BotVisit + Page (upsert)
```

1. Cloudflare Worker intercepts requests
2. Classification Engine detects AI bot user agents
3. Data sent to `/api/bots`
4. Raw data stored in CrawlerLog (for debugging)
5. Processed data stored in BotVisit (for analytics)
6. Page statistics updated

### Cleanup Flow

```
Cron Job → /api/maintenance/cleanup → CrawlerLog (delete old entries)
```

1. Daily cron job triggers cleanup
2. Entries older than 30 days removed from CrawlerLog
3. Analytics tables remain intact

## Core Components

### Tracker Package

Located in `packages/tracker/`:

```
tracker/
├── src/
│   ├── core/tracker.ts      # Main tracker
│   ├── session/session.ts   # Session management
│   ├── source/ai-source.ts  # AI detection
│   └── events/              # Event types
└── build.ts                 # Build script
```

**Features:**
- AI source detection (referrer-based)
- Session tracking
- Event buffering
- SPA support

### AI Classification Engine

Located in `src/lib/ai-classification-engine.ts`:

**Features:**
- Multi-factor analysis (User-Agent, Referer, Headers)
- Business-oriented categorization
- Hot-updatable rules via JSON config
- Confidence scoring

**Classification Categories:**
| Category | Description | Examples |
|----------|-------------|----------|
| AI_CRAWLER | Training crawlers | GPTBot, ClaudeBot |
| AI_SEARCH_CRAWLER | Search engine AI | OAI-SearchBot |
| AI_USER_FETCH | User-initiated | ChatGPT-User |
| AI_REFERRAL | Chat interface traffic | chatgpt.com |
| AI_BROWSER_AGENT | Browser-based AI | OpenAI Operator |

### Dashboard

Located in `src/app/dashboard/`:

| Page | Purpose |
|------|---------|
| Overview | Activity feed and key stats |
| Pages | Page discovery and performance |
| Bots | AI crawler monitoring |
| Referrals | AI referral traffic analysis |
| Settings | Account configuration |

### API Routes

Located in `src/app/api/`:

| Route | Purpose | Tables Used |
|-------|---------|-------------|
| /tracker | Collect user events | Session, Event, Referral |
| /bots | Bot activity data | BotVisit, Page |
| /pages | Page discovery data | Page |
| /referrals | Referral analytics | Referral |
| /overview | Dashboard statistics | BotVisit, Referral, Page |
| /stripe | Payment webhooks | Subscription |
| /maintenance/cleanup | Data cleanup | CrawlerLog |

## Security

### Authentication
- Clerk handles user authentication
- Session-based auth with JWT tokens
- API keys for Agency plan

### Data Protection
- No personal data collected
- IP addresses hashed (optional)
- HTTPS required for all requests

### API Security
- Rate limiting on public endpoints
- API keys for Agency plan
- CORS restrictions

## Performance

### Frontend
- Server Components for initial load
- Client Components for interactivity
- Static page generation where possible

### Backend
- Connection pooling for database
- Indexed queries for common patterns
- Caching for dashboard stats
- Raw logs isolated from analytics queries

## Scalability

### Current Architecture
- Single database instance
- Serverless functions (Vercel)
- Built-in read scaling via Vercel Edge Network

### Future Scaling
- Read replicas for analytics queries
- Queue-based event processing
- CDN for tracker script
- Partitioning by siteId for large datasets

## Monitoring

### Health Check
- `/api/health` endpoint available

### Logging
- Structured logging throughout
- Error tracking integration ready

### Metrics
- Dashboard for system health
- Query performance monitoring

---

Next: [Database Schema](./database-schema.md)