diff --git a/docs/CRM_AND_WORKFLOW_BENCHMARKS.md b/docs/CRM_AND_WORKFLOW_BENCHMARKS.md new file mode 100644 index 0000000..36a7ba0 --- /dev/null +++ b/docs/CRM_AND_WORKFLOW_BENCHMARKS.md @@ -0,0 +1,99 @@ +# OpenForce Benchmark Analysis: CRM Architectures, Custom Object Engines & Visual Workflow Editors + +## 1. Executive Summary +This document establishes architectural benchmarks, technical design trade-offs, and proven implementation patterns for **OpenForce**—a Python/Django + Vue 3 open-source CRM targeting Salesforce parity. + +Key benchmark insights: +1. **Custom Object Storage**: Salesforce utilizes a multi-tenant universal table architecture (MT_Objects, MT_Fields, MT_Value_Data). Modern PostgreSQL allows us to achieve superior developer ergonomics and query performance via a **Hybrid JSONB + Typed Auxiliary Indexes** architecture (as proven by Twenty CRM and Supabase dynamic schema patterns). +2. **Visual Workflow Canvas**: `@vue-flow/core` provides native reactivity and customizable node sockets that match the UX paradigms of Salesforce Flow Builder, Zapier, and n8n. +3. **Execution Runtime**: A lightweight Directed Acyclic Graph (DAG) state machine running atop Celery/Redis guarantees idempotency, branch isolation, and transactional step retries. + +--- + +## 2. Comparative Analysis of Open-Source CRM Architectures + +| Dimension | Salesforce (Proprietary) | Twenty CRM (TypeScript/Nest/Postgres) | SuiteCRM 8 (PHP/Symfony) | OpenForce (Proposed: Django/Vue) | +| :--- | :--- | :--- | :--- | :--- | +| **Object Engine** | Proprietary Force.com multi-tenant UDM (EAV variant) | Dynamic PostgreSQL tables generated per workspace via TypeORM metadata | Traditional relational EAV (`*_cstm` tables per module) | **Hybrid PostgreSQL JSONB with GIN index + Metamodel Validation** | +| **Schema Flexibility** | High (Custom fields, relationships, validation rules) | High (Dynamic schema migrations per workspace) | Medium (Alters database schema on field creation) | **High (Instant zero-migration schema updates via JSONB)** | +| **Workflow Engine** | Salesforce Flow / Process Builder (Apex engine) | Action trigger workflows (Server-side TS) | Logic Hooks + WorkFlow module (Cron/Event triggers) | **DAG Execution Engine with Celery task orchestration** | +| **Frontend Framework** | Lightning Web Components (LWC) | React + Tailwind + Vite | Angular + Legacy Smarty/Vue | **Vue 3 (Composition API) + Pinia + TailwindCSS** | +| **API Architecture** | REST / SOAP / GraphQL / Bulk API | GraphQL first + REST | REST API v8 (JSON:API spec) | **Django REST Framework (DRF) + OpenAPI v3 / Dynamic ViewSets** | + +--- + +## 3. Deep-Dive: Custom Object & Dynamic Field Engine + +Salesforce's Force.com architecture avoids running `ALTER TABLE` whenever an administrator creates a custom object or field. Instead, it relies on metadata tables describing tenant fields and a massive denormalized storage layer. + +### 3.1 Evaluated Architecture Patterns + +#### Pattern A: Strict EAV (Entity-Attribute-Value) +- **Mechanism**: Table `CustomValues (record_id, field_id, value_string, value_int, ...)`. +- **Pros**: Pure relational modeling, strict SQL types. +- **Cons**: Massive JOIN overhead when querying 20+ fields per record; poor sorting/filtering performance across dynamic fields. + +#### Pattern B: Dynamic DDL (`ALTER TABLE` on the fly) +- **Mechanism**: Django dynamically creates and executes `ALTER TABLE my_table ADD COLUMN custom_field VARCHAR`. +- **Pros**: Native DB columns, native indexes, simple SQL. +- **Cons**: DB lock contention, DDL transaction risks, migration tracking complexity across production instances. + +#### Pattern C: Hybrid JSONB + Metadata Metamodel (Recommended for OpenForce) +- **Mechanism**: + - Standard relational tables for fixed objects (`Account`, `Contact`, `Lead`, `Opportunity`). + - Metamodel tables: `CustomObjectDefinition` and `CustomFieldDefinition`. + - Data table: `CustomRecord` with `(id, object_definition_id, data JSONB, created_at, updated_at, created_by)`. + - Native PostgreSQL GIN indexes on `data`: `CREATE INDEX idx_custom_record_gin ON custom_records USING gin (data jsonb_path_ops);`. + - Dynamic virtual serialization in DRF with JSONSchema validation at ingestion. +- **Pros**: + - Zero-downtime dynamic schema updates. + - Sub-millisecond JSONB query performance via GIN indexing and expression indexes for frequent filters (e.g. `CREATE INDEX idx_status ON custom_records ((data->>'status'));`). + - Seamless support for all Salesforce data types (picklist, multiselect, lookup, currency, formula). + +--- + +## 4. Visual Workflow Automation Engine Benchmark + +### 4.1 Frontend Canvas Benchmark: Vue Flow vs. Alternatives +- **Selected: `@vue-flow/core` (v1.30+)**: + - Direct Vue 3 port inspired by React Flow. + - Native Vue reactivity: Custom node components can be pure `.vue` Single File Components (SFCs). + - Out-of-the-box support for interactive controls, minimap, background grid, smooth bezier edges, and multi-socket connection validation. + - UX Pattern matches Salesforce Flow Builder: + - **Palette Drawer**: Drag triggers, logic nodes, and action blocks into viewport. + - **Step Inspector Side-Panel**: Contextual form for editing trigger conditions or email action parameters when a node is selected. + - **Live Graph Validation**: Visual feedback if branches are orphaned or cycles are detected. + +### 4.2 Backend Execution Runtime (DAG Engine) +- **Workflow Representation**: Stored as a JSON DAG: + ```json + { + "nodes": [ + {"id": "node_1", "type": "trigger", "data": {"event": "lead.created"}}, + {"id": "node_2", "type": "condition", "data": {"rules": [{"field": "lead_score", "op": "gte", "value": 50}]}}, + {"id": "node_3", "type": "action", "data": {"action": "email.send_template", "template_id": 4}} + ], + "edges": [ + {"source": "node_1", "target": "node_2"}, + {"source": "node_2", "target": "node_3", "sourceHandle": "true"} + ] + } + ``` +- **Execution Model**: + - Synchronous / Celery worker dispatches flow runs on Django model signals (`post_save`). + - Topologically sorted step resolution with branch isolation. + - Idempotency keys generated per event trigger to prevent duplicate execution. + +--- + +## 5. Standard CRM Life-Cycle Recommendations + +1. **Lead Conversion Flow**: + - Atomic transaction converting `Lead(status='Qualified')` -> creates `Account` (or merges into existing), creates `Contact`, and optionally creates `Opportunity`. + - Copies custom fields from Lead to matching target fields on Account/Contact/Opportunity via field mapping metadata. +2. **Polymorphic Activity Timeline**: + - Django `GenericForeignKey` to associate `Tasks`, `Notes`, `Emails`, and `StageHistory` with any model (`Lead`, `Opportunity`, or `CustomRecord`). +3. **Email Outreach Engine**: + - Queue-backed (Celery) bulk sending with rate-limiting. + - Template engine using Django templating or Jinja2 syntax (`{{ contact.first_name }}`) with fallback defaults. + diff --git a/docs/STANDARD_CRM_SCHEMA_SPEC.md b/docs/STANDARD_CRM_SCHEMA_SPEC.md new file mode 100644 index 0000000..d89e009 --- /dev/null +++ b/docs/STANDARD_CRM_SCHEMA_SPEC.md @@ -0,0 +1,151 @@ +# OpenForce Standard CRM Schema Specification & Field Dictionary +**Target Entities**: `Account`, `Contact`, `Lead`, `Opportunity`, and `Activity` + +This document defines the production field dictionary, constraint rules, indexing strategies, and lifecycle conversion dynamics for OpenForce's Phase 1 standard objects, benchmarking Salesforce standard fields, Twenty CRM, and SuiteCRM. + +--- + +## 1. Account (Standard Object) +Represents a business organization, partner, or enterprise client. + +### Fields & Constraints +| Field Name | Type | Constraints | Description | +| :--- | :--- | :--- | :--- | +| `id` | UUID | Primary Key, default `uuid.uuid4` | Global unique identifier | +| `name` | String(255) | Required, indexed | Company or organization legal/trading name | +| `account_type` | Enum / String(50) | Choices: `Customer`, `Prospect`, `Partner`, `Competitor`, `Vendor`, `Other` | Relationship type | +| `industry` | Enum / String(100) | Choices: `Technology`, `Finance`, `Healthcare`, `Manufacturing`, `Retail`, `Services`, `Other` | Sector categorization | +| `annual_revenue` | Decimal(18, 2) | Nullable, default `None` | Annual estimated or reported turnover | +| `employees_count`| Integer | Nullable, min 0 | Total workforce count | +| `website` | URLField | Nullable, max_length 255 | Corporate website URL | +| `phone` | String(40) | Nullable | Primary corporate telephone number | +| `billing_address`| JSONB | Schema: `{street, city, state, postal_code, country}` | Structured billing address | +| `shipping_address`| JSONB | Schema: `{street, city, state, postal_code, country}` | Structured shipping/logistics address | +| `parent_account` | ForeignKey(Account) | Nullable, `on_delete=SET_NULL`, related_name=`child_accounts` | Corporate hierarchy/subsidiary tree | +| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`accounts` | Account executive / sales rep owner | +| `created_at` | DateTime | Auto now add, indexed | Creation timestamp | +| `updated_at` | DateTime | Auto now, indexed | Last updated timestamp | + +--- + +## 2. Contact (Standard Object) +Represents an individual person associated with an Account. + +### Fields & Constraints +| Field Name | Type | Constraints | Description | +| :--- | :--- | :--- | :--- | +| `id` | UUID | Primary Key | Global unique identifier | +| `account` | ForeignKey(Account) | Nullable, `on_delete=CASCADE`, related_name=`contacts`, indexed | Linked company/account | +| `first_name` | String(100) | Required | Individual's given name | +| `last_name` | String(100) | Required, indexed | Individual's family / surname | +| `email` | EmailField | Indexed, nullable | Primary email address | +| `phone` | String(40) | Nullable | Direct business telephone | +| `mobile_phone` | String(40) | Nullable | Mobile/cellular telephone | +| `title` | String(128) | Nullable | Job title / professional role (e.g. VP Engineering) | +| `department` | String(128) | Nullable | Department within organization | +| `is_primary` | Boolean | Default `False` | Primary contact flag for the parent Account | +| `do_not_call` | Boolean | Default `False` | Compliance flag for outbound calling | +| `email_opt_out` | Boolean | Default `False`, indexed | Unsubscribe flag for automated outreach campaigns | +| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`contacts` | Relationship manager | +| `created_at` | DateTime | Auto now add | Creation timestamp | +| `updated_at` | DateTime | Auto now | Last updated timestamp | + +--- + +## 3. Lead (Standard Object) +Represents an unqualified prospect or inbound marketing inquiry. + +### Fields & Constraints +| Field Name | Type | Constraints | Description | +| :--- | :--- | :--- | :--- | +| `id` | UUID | Primary Key | Global unique identifier | +| `first_name` | String(100) | Required | Prospect given name | +| `last_name` | String(100) | Required, indexed | Prospect family name | +| `company` | String(255) | Required, indexed | Organization name provided by prospect | +| `email` | EmailField | Indexed, nullable | Prospect email address | +| `phone` | String(40) | Nullable | Phone number | +| `lead_source` | Enum / String(64) | Choices: `Inbound Web`, `Referral`, `Conference`, `Cold Outreach`, `Partner`, `Other` | Acquisition origin | +| `status` | Enum / String(50) | Choices: `New`, `Contacted`, `Working`, `Qualified`, `Unqualified` | Lead lifecycle status | +| `lead_score` | Integer | Default 0, indexed | Heuristic engagement score (0-100) | +| `is_converted` | Boolean | Default `False`, indexed | True if converted to Account/Contact/Opp | +| `converted_at` | DateTime | Nullable | Conversion timestamp | +| `converted_account` | ForeignKey(Account) | Nullable, `on_delete=SET_NULL` | Account generated during conversion | +| `converted_contact` | ForeignKey(Contact) | Nullable, `on_delete=SET_NULL` | Contact generated during conversion | +| `converted_opportunity` | ForeignKey(Opportunity) | Nullable, `on_delete=SET_NULL` | Deal created during conversion | +| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`leads` | Sales development representative | +| `created_at` | DateTime | Auto now add | Creation timestamp | +| `updated_at` | DateTime | Auto now | Last updated timestamp | + +--- + +## 4. Opportunity (Standard Object) +Represents a prospective deal or revenue-generating engagement in the sales pipeline. + +### Fields & Constraints +| Field Name | Type | Constraints | Description | +| :--- | :--- | :--- | :--- | +| `id` | UUID | Primary Key | Global unique identifier | +| `name` | String(255) | Required, indexed | Deal name (e.g. "Acme Corp - 50 Seat License") | +| `account` | ForeignKey(Account) | Required, `on_delete=CASCADE`, related_name=`opportunities` | Associated client account | +| `primary_contact`| ForeignKey(Contact) | Nullable, `on_delete=SET_NULL` | Lead champion / signatory contact | +| `amount` | Decimal(18, 2) | Default `0.00`, indexed | Total deal value in base currency | +| `stage` | Enum / String(50) | Indexed | Pipeline stage (see Stage Dictionary below) | +| `probability` | Integer | Min 0, Max 100 | Percentage probability of winning (stage-derived default) | +| `expected_revenue`| Decimal(18, 2) | Computed (`amount * probability / 100`) | Weighted pipeline revenue | +| `close_date` | Date | Required, indexed | Anticipated or actual transaction close date | +| `lead_source` | Enum / String(64) | Nullable | Origination channel | +| `deal_type` | Enum / String(50) | Choices: `New Business`, `Existing Business - Expansion`, `Renewal` | Commercial categorization | +| `next_step` | String(255) | Nullable | Immediate operational action step | +| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`opportunities` | Deal lead | +| `created_at` | DateTime | Auto now add | Creation timestamp | +| `updated_at` | DateTime | Auto now | Last updated timestamp | + +### Opportunity Stage & Probability Matrix +| Stage Name | Default Probability | Forecast Category | Description | +| :--- | :--- | :--- | :--- | +| `Prospecting` | 10% | Pipeline | Initial contact made, interest identified | +| `Qualification`| 20% | Pipeline | BANT (Budget, Authority, Need, Timeline) assessed | +| `Needs Analysis`| 40% | Pipeline | Deep technical & requirements discovery | +| `Proposal / Quote`| 60% | Best Case | Formal quote or proposal delivered | +| `Negotiation / Review`| 80% | Commit | Legal, procurement, and terms discussion | +| `Closed Won` | 100% | Closed | Executed agreement, deal won | +| `Closed Lost` | 0% | Omitted | Deal abandoned or lost to competitor | + +--- + +## 5. Unified Activity & Task Model +Tracks all activities, calls, emails, and to-dos against standard and custom entities. + +### Activity / Task Schema +| Field Name | Type | Constraints | Description | +| :--- | :--- | :--- | :--- | +| `id` | UUID | Primary Key | Global unique identifier | +| `subject` | String(255) | Required | Activity headline / task title | +| `activity_type` | Enum / String(32) | Choices: `Task`, `Call`, `Meeting`, `Email`, `Note` | Classification of interaction | +| `status` | Enum / String(32) | Choices: `Not Started`, `In Progress`, `Completed`, `Deferred` | Execution status | +| `priority` | Enum / String(20) | Choices: `High`, `Normal`, `Low` | Urgency tier | +| `due_date` | DateTime | Nullable, indexed | Target completion or occurrence time | +| `description` | TextField | Nullable | Body notes or meeting transcript | +| `content_type` | ForeignKey(ContentType) | Indexed | Generic relation to any Django model | +| `object_id` | String(64) | Indexed | ID of target entity (Account, Contact, Lead, etc.) | +| `content_object`| GenericForeignKey | Resolves `(content_type, object_id)` | Polymorphic target link | +| `assigned_to` | ForeignKey(User) | Nullable, `on_delete=SET_NULL` | Action assignee | +| `completed_at` | DateTime | Nullable | Actual completion timestamp | + +--- + +## 6. Lead Conversion Pipeline Mechanics +When a user or automated flow invokes `POST /api/leads/{id}/convert/`: +1. **Validation**: Verify `lead.is_converted == False`. +2. **Atomic Transaction (`transaction.atomic`)**: + - Locate or create `Account` matching `lead.company`. + - Create `Contact` with `first_name`, `last_name`, `email`, `phone`, linked to `Account`. + - If requested, create `Opportunity` with `name=f"{lead.company} - New Deal"`, `stage="Prospecting"`, linked to `Account` and `Contact`. + - Update `Lead`: + - `is_converted = True` + - `converted_at = timezone.now()` + - `converted_account = account` + - `converted_contact = contact` + - `converted_opportunity = opportunity` + - Re-link all related polymorphic `Tasks` and `Activities` from `(Lead, lead.id)` to `(Contact, contact.id)` or `(Account, account.id)`. +