docs: add CRM benchmarks and standard schema specifications
This commit is contained in:
parent
f51055cea5
commit
929845fb9a
2 changed files with 250 additions and 0 deletions
99
docs/CRM_AND_WORKFLOW_BENCHMARKS.md
Normal file
99
docs/CRM_AND_WORKFLOW_BENCHMARKS.md
Normal file
|
|
@ -0,0 +1,99 @@
|
||||||
|
# OpenForce Benchmark Analysis: CRM Architectures, Custom Object Engines & Visual Workflow Editors
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
This document establishes architectural benchmarks, technical design trade-offs, and proven implementation patterns for **OpenForce**—a Python/Django + Vue 3 open-source CRM targeting Salesforce parity.
|
||||||
|
|
||||||
|
Key benchmark insights:
|
||||||
|
1. **Custom Object Storage**: Salesforce utilizes a multi-tenant universal table architecture (MT_Objects, MT_Fields, MT_Value_Data). Modern PostgreSQL allows us to achieve superior developer ergonomics and query performance via a **Hybrid JSONB + Typed Auxiliary Indexes** architecture (as proven by Twenty CRM and Supabase dynamic schema patterns).
|
||||||
|
2. **Visual Workflow Canvas**: `@vue-flow/core` provides native reactivity and customizable node sockets that match the UX paradigms of Salesforce Flow Builder, Zapier, and n8n.
|
||||||
|
3. **Execution Runtime**: A lightweight Directed Acyclic Graph (DAG) state machine running atop Celery/Redis guarantees idempotency, branch isolation, and transactional step retries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Comparative Analysis of Open-Source CRM Architectures
|
||||||
|
|
||||||
|
| Dimension | Salesforce (Proprietary) | Twenty CRM (TypeScript/Nest/Postgres) | SuiteCRM 8 (PHP/Symfony) | OpenForce (Proposed: Django/Vue) |
|
||||||
|
| :--- | :--- | :--- | :--- | :--- |
|
||||||
|
| **Object Engine** | Proprietary Force.com multi-tenant UDM (EAV variant) | Dynamic PostgreSQL tables generated per workspace via TypeORM metadata | Traditional relational EAV (`*_cstm` tables per module) | **Hybrid PostgreSQL JSONB with GIN index + Metamodel Validation** |
|
||||||
|
| **Schema Flexibility** | High (Custom fields, relationships, validation rules) | High (Dynamic schema migrations per workspace) | Medium (Alters database schema on field creation) | **High (Instant zero-migration schema updates via JSONB)** |
|
||||||
|
| **Workflow Engine** | Salesforce Flow / Process Builder (Apex engine) | Action trigger workflows (Server-side TS) | Logic Hooks + WorkFlow module (Cron/Event triggers) | **DAG Execution Engine with Celery task orchestration** |
|
||||||
|
| **Frontend Framework** | Lightning Web Components (LWC) | React + Tailwind + Vite | Angular + Legacy Smarty/Vue | **Vue 3 (Composition API) + Pinia + TailwindCSS** |
|
||||||
|
| **API Architecture** | REST / SOAP / GraphQL / Bulk API | GraphQL first + REST | REST API v8 (JSON:API spec) | **Django REST Framework (DRF) + OpenAPI v3 / Dynamic ViewSets** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Deep-Dive: Custom Object & Dynamic Field Engine
|
||||||
|
|
||||||
|
Salesforce's Force.com architecture avoids running `ALTER TABLE` whenever an administrator creates a custom object or field. Instead, it relies on metadata tables describing tenant fields and a massive denormalized storage layer.
|
||||||
|
|
||||||
|
### 3.1 Evaluated Architecture Patterns
|
||||||
|
|
||||||
|
#### Pattern A: Strict EAV (Entity-Attribute-Value)
|
||||||
|
- **Mechanism**: Table `CustomValues (record_id, field_id, value_string, value_int, ...)`.
|
||||||
|
- **Pros**: Pure relational modeling, strict SQL types.
|
||||||
|
- **Cons**: Massive JOIN overhead when querying 20+ fields per record; poor sorting/filtering performance across dynamic fields.
|
||||||
|
|
||||||
|
#### Pattern B: Dynamic DDL (`ALTER TABLE` on the fly)
|
||||||
|
- **Mechanism**: Django dynamically creates and executes `ALTER TABLE my_table ADD COLUMN custom_field VARCHAR`.
|
||||||
|
- **Pros**: Native DB columns, native indexes, simple SQL.
|
||||||
|
- **Cons**: DB lock contention, DDL transaction risks, migration tracking complexity across production instances.
|
||||||
|
|
||||||
|
#### Pattern C: Hybrid JSONB + Metadata Metamodel (Recommended for OpenForce)
|
||||||
|
- **Mechanism**:
|
||||||
|
- Standard relational tables for fixed objects (`Account`, `Contact`, `Lead`, `Opportunity`).
|
||||||
|
- Metamodel tables: `CustomObjectDefinition` and `CustomFieldDefinition`.
|
||||||
|
- Data table: `CustomRecord` with `(id, object_definition_id, data JSONB, created_at, updated_at, created_by)`.
|
||||||
|
- Native PostgreSQL GIN indexes on `data`: `CREATE INDEX idx_custom_record_gin ON custom_records USING gin (data jsonb_path_ops);`.
|
||||||
|
- Dynamic virtual serialization in DRF with JSONSchema validation at ingestion.
|
||||||
|
- **Pros**:
|
||||||
|
- Zero-downtime dynamic schema updates.
|
||||||
|
- Sub-millisecond JSONB query performance via GIN indexing and expression indexes for frequent filters (e.g. `CREATE INDEX idx_status ON custom_records ((data->>'status'));`).
|
||||||
|
- Seamless support for all Salesforce data types (picklist, multiselect, lookup, currency, formula).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Visual Workflow Automation Engine Benchmark
|
||||||
|
|
||||||
|
### 4.1 Frontend Canvas Benchmark: Vue Flow vs. Alternatives
|
||||||
|
- **Selected: `@vue-flow/core` (v1.30+)**:
|
||||||
|
- Direct Vue 3 port inspired by React Flow.
|
||||||
|
- Native Vue reactivity: Custom node components can be pure `.vue` Single File Components (SFCs).
|
||||||
|
- Out-of-the-box support for interactive controls, minimap, background grid, smooth bezier edges, and multi-socket connection validation.
|
||||||
|
- UX Pattern matches Salesforce Flow Builder:
|
||||||
|
- **Palette Drawer**: Drag triggers, logic nodes, and action blocks into viewport.
|
||||||
|
- **Step Inspector Side-Panel**: Contextual form for editing trigger conditions or email action parameters when a node is selected.
|
||||||
|
- **Live Graph Validation**: Visual feedback if branches are orphaned or cycles are detected.
|
||||||
|
|
||||||
|
### 4.2 Backend Execution Runtime (DAG Engine)
|
||||||
|
- **Workflow Representation**: Stored as a JSON DAG:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"nodes": [
|
||||||
|
{"id": "node_1", "type": "trigger", "data": {"event": "lead.created"}},
|
||||||
|
{"id": "node_2", "type": "condition", "data": {"rules": [{"field": "lead_score", "op": "gte", "value": 50}]}},
|
||||||
|
{"id": "node_3", "type": "action", "data": {"action": "email.send_template", "template_id": 4}}
|
||||||
|
],
|
||||||
|
"edges": [
|
||||||
|
{"source": "node_1", "target": "node_2"},
|
||||||
|
{"source": "node_2", "target": "node_3", "sourceHandle": "true"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **Execution Model**:
|
||||||
|
- Synchronous / Celery worker dispatches flow runs on Django model signals (`post_save`).
|
||||||
|
- Topologically sorted step resolution with branch isolation.
|
||||||
|
- Idempotency keys generated per event trigger to prevent duplicate execution.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Standard CRM Life-Cycle Recommendations
|
||||||
|
|
||||||
|
1. **Lead Conversion Flow**:
|
||||||
|
- Atomic transaction converting `Lead(status='Qualified')` -> creates `Account` (or merges into existing), creates `Contact`, and optionally creates `Opportunity`.
|
||||||
|
- Copies custom fields from Lead to matching target fields on Account/Contact/Opportunity via field mapping metadata.
|
||||||
|
2. **Polymorphic Activity Timeline**:
|
||||||
|
- Django `GenericForeignKey` to associate `Tasks`, `Notes`, `Emails`, and `StageHistory` with any model (`Lead`, `Opportunity`, or `CustomRecord`).
|
||||||
|
3. **Email Outreach Engine**:
|
||||||
|
- Queue-backed (Celery) bulk sending with rate-limiting.
|
||||||
|
- Template engine using Django templating or Jinja2 syntax (`{{ contact.first_name }}`) with fallback defaults.
|
||||||
|
|
||||||
151
docs/STANDARD_CRM_SCHEMA_SPEC.md
Normal file
151
docs/STANDARD_CRM_SCHEMA_SPEC.md
Normal file
|
|
@ -0,0 +1,151 @@
|
||||||
|
# OpenForce Standard CRM Schema Specification & Field Dictionary
|
||||||
|
**Target Entities**: `Account`, `Contact`, `Lead`, `Opportunity`, and `Activity`
|
||||||
|
|
||||||
|
This document defines the production field dictionary, constraint rules, indexing strategies, and lifecycle conversion dynamics for OpenForce's Phase 1 standard objects, benchmarking Salesforce standard fields, Twenty CRM, and SuiteCRM.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Account (Standard Object)
|
||||||
|
Represents a business organization, partner, or enterprise client.
|
||||||
|
|
||||||
|
### Fields & Constraints
|
||||||
|
| Field Name | Type | Constraints | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `id` | UUID | Primary Key, default `uuid.uuid4` | Global unique identifier |
|
||||||
|
| `name` | String(255) | Required, indexed | Company or organization legal/trading name |
|
||||||
|
| `account_type` | Enum / String(50) | Choices: `Customer`, `Prospect`, `Partner`, `Competitor`, `Vendor`, `Other` | Relationship type |
|
||||||
|
| `industry` | Enum / String(100) | Choices: `Technology`, `Finance`, `Healthcare`, `Manufacturing`, `Retail`, `Services`, `Other` | Sector categorization |
|
||||||
|
| `annual_revenue` | Decimal(18, 2) | Nullable, default `None` | Annual estimated or reported turnover |
|
||||||
|
| `employees_count`| Integer | Nullable, min 0 | Total workforce count |
|
||||||
|
| `website` | URLField | Nullable, max_length 255 | Corporate website URL |
|
||||||
|
| `phone` | String(40) | Nullable | Primary corporate telephone number |
|
||||||
|
| `billing_address`| JSONB | Schema: `{street, city, state, postal_code, country}` | Structured billing address |
|
||||||
|
| `shipping_address`| JSONB | Schema: `{street, city, state, postal_code, country}` | Structured shipping/logistics address |
|
||||||
|
| `parent_account` | ForeignKey(Account) | Nullable, `on_delete=SET_NULL`, related_name=`child_accounts` | Corporate hierarchy/subsidiary tree |
|
||||||
|
| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`accounts` | Account executive / sales rep owner |
|
||||||
|
| `created_at` | DateTime | Auto now add, indexed | Creation timestamp |
|
||||||
|
| `updated_at` | DateTime | Auto now, indexed | Last updated timestamp |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Contact (Standard Object)
|
||||||
|
Represents an individual person associated with an Account.
|
||||||
|
|
||||||
|
### Fields & Constraints
|
||||||
|
| Field Name | Type | Constraints | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `id` | UUID | Primary Key | Global unique identifier |
|
||||||
|
| `account` | ForeignKey(Account) | Nullable, `on_delete=CASCADE`, related_name=`contacts`, indexed | Linked company/account |
|
||||||
|
| `first_name` | String(100) | Required | Individual's given name |
|
||||||
|
| `last_name` | String(100) | Required, indexed | Individual's family / surname |
|
||||||
|
| `email` | EmailField | Indexed, nullable | Primary email address |
|
||||||
|
| `phone` | String(40) | Nullable | Direct business telephone |
|
||||||
|
| `mobile_phone` | String(40) | Nullable | Mobile/cellular telephone |
|
||||||
|
| `title` | String(128) | Nullable | Job title / professional role (e.g. VP Engineering) |
|
||||||
|
| `department` | String(128) | Nullable | Department within organization |
|
||||||
|
| `is_primary` | Boolean | Default `False` | Primary contact flag for the parent Account |
|
||||||
|
| `do_not_call` | Boolean | Default `False` | Compliance flag for outbound calling |
|
||||||
|
| `email_opt_out` | Boolean | Default `False`, indexed | Unsubscribe flag for automated outreach campaigns |
|
||||||
|
| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`contacts` | Relationship manager |
|
||||||
|
| `created_at` | DateTime | Auto now add | Creation timestamp |
|
||||||
|
| `updated_at` | DateTime | Auto now | Last updated timestamp |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Lead (Standard Object)
|
||||||
|
Represents an unqualified prospect or inbound marketing inquiry.
|
||||||
|
|
||||||
|
### Fields & Constraints
|
||||||
|
| Field Name | Type | Constraints | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `id` | UUID | Primary Key | Global unique identifier |
|
||||||
|
| `first_name` | String(100) | Required | Prospect given name |
|
||||||
|
| `last_name` | String(100) | Required, indexed | Prospect family name |
|
||||||
|
| `company` | String(255) | Required, indexed | Organization name provided by prospect |
|
||||||
|
| `email` | EmailField | Indexed, nullable | Prospect email address |
|
||||||
|
| `phone` | String(40) | Nullable | Phone number |
|
||||||
|
| `lead_source` | Enum / String(64) | Choices: `Inbound Web`, `Referral`, `Conference`, `Cold Outreach`, `Partner`, `Other` | Acquisition origin |
|
||||||
|
| `status` | Enum / String(50) | Choices: `New`, `Contacted`, `Working`, `Qualified`, `Unqualified` | Lead lifecycle status |
|
||||||
|
| `lead_score` | Integer | Default 0, indexed | Heuristic engagement score (0-100) |
|
||||||
|
| `is_converted` | Boolean | Default `False`, indexed | True if converted to Account/Contact/Opp |
|
||||||
|
| `converted_at` | DateTime | Nullable | Conversion timestamp |
|
||||||
|
| `converted_account` | ForeignKey(Account) | Nullable, `on_delete=SET_NULL` | Account generated during conversion |
|
||||||
|
| `converted_contact` | ForeignKey(Contact) | Nullable, `on_delete=SET_NULL` | Contact generated during conversion |
|
||||||
|
| `converted_opportunity` | ForeignKey(Opportunity) | Nullable, `on_delete=SET_NULL` | Deal created during conversion |
|
||||||
|
| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`leads` | Sales development representative |
|
||||||
|
| `created_at` | DateTime | Auto now add | Creation timestamp |
|
||||||
|
| `updated_at` | DateTime | Auto now | Last updated timestamp |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Opportunity (Standard Object)
|
||||||
|
Represents a prospective deal or revenue-generating engagement in the sales pipeline.
|
||||||
|
|
||||||
|
### Fields & Constraints
|
||||||
|
| Field Name | Type | Constraints | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `id` | UUID | Primary Key | Global unique identifier |
|
||||||
|
| `name` | String(255) | Required, indexed | Deal name (e.g. "Acme Corp - 50 Seat License") |
|
||||||
|
| `account` | ForeignKey(Account) | Required, `on_delete=CASCADE`, related_name=`opportunities` | Associated client account |
|
||||||
|
| `primary_contact`| ForeignKey(Contact) | Nullable, `on_delete=SET_NULL` | Lead champion / signatory contact |
|
||||||
|
| `amount` | Decimal(18, 2) | Default `0.00`, indexed | Total deal value in base currency |
|
||||||
|
| `stage` | Enum / String(50) | Indexed | Pipeline stage (see Stage Dictionary below) |
|
||||||
|
| `probability` | Integer | Min 0, Max 100 | Percentage probability of winning (stage-derived default) |
|
||||||
|
| `expected_revenue`| Decimal(18, 2) | Computed (`amount * probability / 100`) | Weighted pipeline revenue |
|
||||||
|
| `close_date` | Date | Required, indexed | Anticipated or actual transaction close date |
|
||||||
|
| `lead_source` | Enum / String(64) | Nullable | Origination channel |
|
||||||
|
| `deal_type` | Enum / String(50) | Choices: `New Business`, `Existing Business - Expansion`, `Renewal` | Commercial categorization |
|
||||||
|
| `next_step` | String(255) | Nullable | Immediate operational action step |
|
||||||
|
| `owner` | ForeignKey(User) | Nullable, `on_delete=SET_NULL`, related_name=`opportunities` | Deal lead |
|
||||||
|
| `created_at` | DateTime | Auto now add | Creation timestamp |
|
||||||
|
| `updated_at` | DateTime | Auto now | Last updated timestamp |
|
||||||
|
|
||||||
|
### Opportunity Stage & Probability Matrix
|
||||||
|
| Stage Name | Default Probability | Forecast Category | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `Prospecting` | 10% | Pipeline | Initial contact made, interest identified |
|
||||||
|
| `Qualification`| 20% | Pipeline | BANT (Budget, Authority, Need, Timeline) assessed |
|
||||||
|
| `Needs Analysis`| 40% | Pipeline | Deep technical & requirements discovery |
|
||||||
|
| `Proposal / Quote`| 60% | Best Case | Formal quote or proposal delivered |
|
||||||
|
| `Negotiation / Review`| 80% | Commit | Legal, procurement, and terms discussion |
|
||||||
|
| `Closed Won` | 100% | Closed | Executed agreement, deal won |
|
||||||
|
| `Closed Lost` | 0% | Omitted | Deal abandoned or lost to competitor |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Unified Activity & Task Model
|
||||||
|
Tracks all activities, calls, emails, and to-dos against standard and custom entities.
|
||||||
|
|
||||||
|
### Activity / Task Schema
|
||||||
|
| Field Name | Type | Constraints | Description |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| `id` | UUID | Primary Key | Global unique identifier |
|
||||||
|
| `subject` | String(255) | Required | Activity headline / task title |
|
||||||
|
| `activity_type` | Enum / String(32) | Choices: `Task`, `Call`, `Meeting`, `Email`, `Note` | Classification of interaction |
|
||||||
|
| `status` | Enum / String(32) | Choices: `Not Started`, `In Progress`, `Completed`, `Deferred` | Execution status |
|
||||||
|
| `priority` | Enum / String(20) | Choices: `High`, `Normal`, `Low` | Urgency tier |
|
||||||
|
| `due_date` | DateTime | Nullable, indexed | Target completion or occurrence time |
|
||||||
|
| `description` | TextField | Nullable | Body notes or meeting transcript |
|
||||||
|
| `content_type` | ForeignKey(ContentType) | Indexed | Generic relation to any Django model |
|
||||||
|
| `object_id` | String(64) | Indexed | ID of target entity (Account, Contact, Lead, etc.) |
|
||||||
|
| `content_object`| GenericForeignKey | Resolves `(content_type, object_id)` | Polymorphic target link |
|
||||||
|
| `assigned_to` | ForeignKey(User) | Nullable, `on_delete=SET_NULL` | Action assignee |
|
||||||
|
| `completed_at` | DateTime | Nullable | Actual completion timestamp |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Lead Conversion Pipeline Mechanics
|
||||||
|
When a user or automated flow invokes `POST /api/leads/{id}/convert/`:
|
||||||
|
1. **Validation**: Verify `lead.is_converted == False`.
|
||||||
|
2. **Atomic Transaction (`transaction.atomic`)**:
|
||||||
|
- Locate or create `Account` matching `lead.company`.
|
||||||
|
- Create `Contact` with `first_name`, `last_name`, `email`, `phone`, linked to `Account`.
|
||||||
|
- If requested, create `Opportunity` with `name=f"{lead.company} - New Deal"`, `stage="Prospecting"`, linked to `Account` and `Contact`.
|
||||||
|
- Update `Lead`:
|
||||||
|
- `is_converted = True`
|
||||||
|
- `converted_at = timezone.now()`
|
||||||
|
- `converted_account = account`
|
||||||
|
- `converted_contact = contact`
|
||||||
|
- `converted_opportunity = opportunity`
|
||||||
|
- Re-link all related polymorphic `Tasks` and `Activities` from `(Lead, lead.id)` to `(Contact, contact.id)` or `(Account, account.id)`.
|
||||||
|
|
||||||
Loading…
Reference in a new issue