openforce/docs/FORMULA_FIELDS_SPEC.md
Bumble 4e64bcf3b4
Some checks failed
Build, Test, and Deploy / build-and-deploy (push) Failing after 29s
docs: add Salesforce Formula Fields and Roll-Up Summaries architectural specification
2026-09-27 04:42:39 +00:00

342 lines
17 KiB
Markdown

# Salesforce Formula Fields & Roll-Up Summaries Architecture Specification
## 1. Executive Summary & Objective
In Salesforce, **Formula Fields** (`Formula(Type)`) and **Roll-Up Summary Fields** (`RollupSummary`) are core declaratively-defined computed metadata types. They enable administrators and developers to calculate values automatically, concatenate strings, evaluate conditional business logic, and aggregate child relationship data without writing imperative Apex code or triggers.
This specification details the architecture for implementing dynamic computed formula fields (`formula`) and roll-up summary aggregations (`rollup_summary`) within OpenForce across:
1. **Dynamic JSONB Custom Fields (`__c`)** on both Standard Entities (`Account`, `Contact`, `Lead`, `Opportunity`, `Activity`, `Campaign`) and Custom Objects (`CustomObject` / `CustomRecord`).
2. **Standard Static Entity Models** via serializer computation and database-level virtual properties.
3. **The AST-based Safe Expression Evaluator** ensuring sandboxed, deterministic runtime computation with zero security risk (forbidding `eval()`, `exec()`, or arbitrary code execution).
4. **Dependency Graph Resolution & Dependency Ordering** ensuring nested formulas (e.g. `formula_b` depending on `formula_a`) evaluate without stale data or infinite loops.
5. **The Unified Object Manager UI/UX** allowing users to choose "Formula" and "Roll-Up Summary" field types, specify return types (`Text`, `Number`, `Currency`, `Percent`, `Date`, `Checkbox`), and test expressions in real time.
---
## 2. Salesforce Formula & Roll-Up Benchmark Analysis
### 2.1 Salesforce Formula Fields vs OpenForce Equivalents
| Capability | Salesforce Specification | OpenForce Implementation |
| :--- | :--- | :--- |
| **Storage Model** | Virtual / calculated at read-time (non-stored), or stored on save | Computed on `pre_save` signal / serializer presentation; persisted in `custom_fields` JSONB for high-speed indexing & query capability |
| **Return Types** | `Checkbox`, `Currency`, `Date`, `DateTime`, `Number`, `Percent`, `Text` | `boolean`, `currency`, `date`, `datetime`, `number`, `percent`, `text` |
| **Expression Syntax** | Excel-like functional syntax: `IF(stage == 'Closed Won', amount * 0.1, 0)` | Excel/Salesforce compatible formula grammar compiled via Python `ast` / token parser |
| **Cross-Object Lookups** | Spanning relations: `Account.Industry`, `Contact.Account.Name` | Dot-notation traversal across Django foreign keys and custom lookup relationships |
| **Standard Operators** | `+`, `-`, `*`, `/`, `^`, `==`, `!=`, `<`, `>`, `<=`, `>=`, `&&`, `\|\|`, `&` (concat) | Full operator support with numeric promotion, safe null coercion, and string concatenation |
| **Built-in Functions** | `IF`, `CASE`, `ISBLANK`, `NOT`, `AND`, `OR`, `TEXT`, `VALUE`, `TODAY`, `NOW`, `CONCAT`, `UPPER`, `LOWER` | Built-in safe standard library module in `custom_objects/formula_engine.py` |
### 2.2 Salesforce Roll-Up Summaries
Salesforce Roll-Up Summaries calculate values from a set of related detail records (Master-Detail relationships):
- **COUNT**: Total number of child records.
- **SUM**: Sum of a specific numeric/currency field across children.
- **MIN / MAX**: Lowest or highest date/number across children.
- **Filter Criteria**: Optional criteria (e.g. `SUM(Opportunity.Amount)` where `Stage == 'Closed Won'`).
In OpenForce:
- Supports reverse relations on standard models (e.g. `Account.opportunities`, `Account.contacts`) and `CustomRecord` child relations.
- Evaluated via Django ORM aggregation (`Sum`, `Count`, `Min`, `Max`) triggered on child `post_save` and `post_delete` signals, atomically updating the parent's `custom_fields['<field_name>']`.
---
## 3. System Architecture & Component Design
```
+-------------------------------------------------------------------------------+
| Unified Object Manager (Vue 3) |
| - Formula Field Creator (Return Type, Expression Editor, Syntax Check) |
| - Roll-Up Summary Creator (Target Child Object, Aggregation Type, Filter) |
+---------------------------------------+---------------------------------------+
| POST /api/custom-fields/
v
+-------------------------------------------------------------------------------+
| Django REST Framework Schema & Validation |
| - Validates formula AST syntax against allowed tokens & referenced fields |
| - Cycle detection in formula dependency graphs (DAG) |
+---------------------------------------+---------------------------------------+
|
+----------------------+-----------------------+
| |
v v
+------------------------------------+ +------------------------------------+
| Formula Evaluation Engine | | Roll-Up Aggregation Engine |
| (`formula_engine.py`) | | (`rollup_engine.py`) |
| - Python AST Safe Visitor | | - Django ORM Signal Triggers |
| - Field Resolver (Local + Lookup) | | - Parent Cache Invalidation |
| - Type Coercion & Formatters | | - Count, Sum, Min, Max Execution |
+-----------------+------------------+ +-----------------+------------------+
| |
+-------------------+--------------------+
|
v
+-------------------------------------------------------------------------------+
| PostgreSQL Storage (`custom_fields` JSONB) |
| - Stored computed values indexed via PostgreSQL GIN indexes |
| - Readily accessible in list views, filters, and workflow automations |
+-------------------------------------------------------------------------------+
```
---
## 4. AST-Based Safe Expression Evaluation Engine
### 4.1 Security Constraints (Strict Sandboxing)
Under **no circumstances** should Python's native `eval()` or `exec()` be used. All expressions are parsed into an Abstract Syntax Tree (`ast.parse(expr, mode='eval')`) and walked using a strict `ast.NodeVisitor`:
1. **Forbidden Nodes**: `Call` to unapproved functions, `Import`, `ImportFrom`, `Lambda`, `ListComp`, `DictComp`, `GeneratorExp`, `Attribute` (unless whitelisted for object traversal), `Subscript`, `Assign`, `Delete`, `While`, `For`.
2. **Allowed Nodes**: `BinOp`, `UnaryOp`, `Compare`, `Name`, `Constant`, `Call` (only against registered whitelist), `Attribute` (for safe relation traversal e.g. `account.name`).
### 4.2 Supported Functions Whitelist
```python
SAFE_FUNCTIONS = {
# Logical
'IF': lambda cond, true_val, false_val: true_val if cond else false_val,
'AND': lambda *args: all(args),
'OR': lambda *args: any(args),
'NOT': lambda x: not bool(x),
'CASE': case_evaluator,
'ISBLANK': lambda val: val is None or val == '' or val == [],
# Text / Formatting
'CONCAT': lambda *args: ''.join(str(a) if a is not None else '' for a in args),
'UPPER': lambda s: str(s or '').upper(),
'LOWER': lambda s: str(s or '').lower(),
'LEN': lambda s: len(str(s or '')),
'TRIM': lambda s: str(s or '').strip(),
'TEXT': lambda v: str(v if v is not None else ''),
'VALUE': lambda s: float(s) if s not in (None, '') else 0.0,
# Math
'ROUND': lambda val, digits=0: round(float(val or 0), int(digits)),
'ABS': lambda val: abs(float(val or 0)),
'MOD': lambda a, b: (float(a) % float(b)) if float(b) != 0 else 0,
'MAX': lambda *args: max(args) if args else None,
'MIN': lambda *args: min(args) if args else None,
'CEILING': lambda val: math.ceil(float(val or 0)),
'FLOOR': lambda val: math.floor(float(val or 0)),
# Date / Time
'TODAY': lambda: timezone.now().date().isoformat(),
'NOW': lambda: timezone.now().isoformat(),
'YEAR': lambda d: parse_date(d).year if d else None,
'MONTH': lambda d: parse_date(d).month if d else None,
'DAY': lambda d: parse_date(d).day if d else None,
}
```
### 4.3 Reference Syntax & Field Resolution
Expressions can reference:
1. **Local Standard Fields**: `amount`, `stage`, `probability`, `first_name`, `annual_revenue`.
2. **Local Custom Fields**: `commission_rate__c`, `discount_percentage__c`.
3. **Cross-Object Relations (Dot notation)**:
- `account.name`
- `account.annual_revenue`
- `contact.account.industry`
---
## 5. Data Model Schema Updates
### 5.1 `CustomFieldDefinition` Field Extensions
To support formulas and rollups, `custom_objects/models.py` extends `CustomFieldDefinition`:
```python
class CustomFieldDefinition(models.Model):
# Existing fields: entity_name, custom_object, name, label, field_type, is_required...
FIELD_TYPE_CHOICES = [
('text', 'Text'),
('number', 'Number'),
('date', 'Date'),
('datetime', 'Date/Time'),
('boolean', 'Checkbox'),
('currency', 'Currency'),
('percent', 'Percent'),
('picklist', 'Picklist'),
('lookup', 'Lookup'),
('formula', 'Formula'), # <--- New
('rollup_summary', 'Roll-Up Summary'), # <--- New
]
# Formula configuration
formula_expression = models.TextField(blank=True, default='')
formula_return_type = models.CharField(
max_length=30,
choices=[
('text', 'Text'),
('number', 'Number'),
('currency', 'Currency'),
('percent', 'Percent'),
('date', 'Date'),
('datetime', 'Date/Time'),
('boolean', 'Checkbox'),
],
blank=True,
null=True
)
formula_decimal_places = models.IntegerField(default=2, blank=True)
# Roll-up summary configuration
rollup_target_entity = models.CharField(max_length=100, blank=True, null=True)
rollup_relationship_field = models.CharField(max_length=100, blank=True, null=True)
rollup_aggregate_type = models.CharField(
max_length=20,
choices=[
('COUNT', 'COUNT'),
('SUM', 'SUM'),
('MIN', 'MIN'),
('MAX', 'MAX'),
],
blank=True,
null=True
)
rollup_aggregate_field = models.CharField(max_length=100, blank=True, null=True)
rollup_filter_criteria = models.JSONField(default=list, blank=True)
```
---
## 6. Execution Lifecycle & Dependency Graph
### 6.1 Dependency Ordering & Cycle Detection
When multiple formula fields exist on an entity (e.g., `net_margin__c` depends on `gross_profit__c`), we resolve dependencies using topological sorting:
```
Record Save / Evaluation Trigger
│
▼
Extract all formula fields for Entity
│
▼
Build Directed Graph of Field References
│
▼
Topological Sort (Detect Cycles -> Raise ValidationError)
│
▼
Evaluate in Topological Order:
Step 1: Evaluate Independent Formulas
Step 2: Inject evaluated values into context
Step 3: Evaluate Dependent Formulas
│
▼
Commit final calculated values to `custom_fields` JSONB
```
### 6.2 Pre-Save Signal Hook (`crm_core/signals.py` & `custom_objects/signals.py`)
A unified `pre_save` signal receiver intercepts record mutations:
```python
@receiver(pre_save)
def evaluate_entity_formula_fields(sender, instance, **kwargs):
entity_name = sender.__name__
if not is_monitored_entity(entity_name):
return
# 1. Fetch all formula definitions for this entity
formula_defs = get_active_formula_fields(entity_name)
if not formula_defs:
return
# 2. Build record evaluation context (standard attributes + custom_fields)
context = build_entity_context(instance)
# 3. Evaluate formulas in topological order
computed_values = evaluate_formulas_topological(formula_defs, context, instance)
# 4. Merge back into custom_fields dictionary
if not instance.custom_fields:
instance.custom_fields = {}
instance.custom_fields.update(computed_values)
```
### 6.3 Roll-Up Summary Signal Hook (`post_save` & `post_delete`)
When child records (e.g. `Opportunity`) are created, updated, or deleted, the parent entity's roll-up summary is recomputed:
```python
@receiver(post_save, sender=Opportunity)
@receiver(post_delete, sender=Opportunity)
def trigger_opportunity_rollup(sender, instance, **kwargs):
if instance.account_id:
recalculate_rollups_for_record('Account', instance.account_id)
```
---
## 7. Frontend Specification: Object Manager Formula & Rollup Experience
### 7.1 Field Creation Wizard Flow (`CustomObjectsView.vue`)
1. **Step 1: Select Field Type**
- User selects `Formula` or `Roll-Up Summary` in the New Field modal.
2. **Step 2: Configure Return Type (for Formula)**
- User chooses: `Text`, `Number`, `Currency`, `Percent`, `Date`, `Checkbox`.
3. **Step 3: Expression Editor**
- Interactive syntax-highlighted textarea with auto-complete pills:
- **Insert Field**: Dropdown of all standard and custom fields on the current entity and lookup parents.
- **Insert Operator**: `+`, `-`, `*`, `/`, `==`, `!=`, `<`, `>`, `&&`, `||`.
- **Insert Function**: `IF()`, `CASE()`, `ISBLANK()`, `CONCAT()`, `ROUND()`, `TODAY()`.
- **"Check Syntax"** button: Triggers `POST /api/schema/validate-formula/` to check AST correctness and surface line-specific error messages before saving.
4. **Step 4: Configure Roll-Up Summary (for Roll-Up)**
- Target Child Object (e.g. `Opportunities`).
- Roll-Up Type (`COUNT`, `SUM`, `MIN`, `MAX`).
- Field to Aggregate (e.g. `amount`).
- Filter criteria (e.g. `stage == 'Closed Won'`).
### 7.2 Unified Table Representation
In the Fields & Relationships grid:
- **Origin Badge**: `Custom (Formula)` or `Custom (Roll-Up)` styled with distinctive indigo and cyan badges.
- **Data Type Column**: Shows `Formula (Currency)`, `Formula (Percent)`, or `Roll-Up Summary (SUM)`.
- **Expression Preview**: Clicking the row displays the read-only formula expression in an inspection drawer.
---
## 8. Sprint Roadmap: Implementation Phases
### Sprint 4.5A: Core Safe AST Formula Engine & Test Suite
- Create `custom_objects/formula_engine.py`:
- AST parser, safe node visitor, token validator.
- Whitelist of functions (`IF`, `CASE`, `ISBLANK`, `CONCAT`, `ROUND`, etc.).
- Cross-object dot notation resolver.
- Topological sorting & cycle detection for chained formulas.
- Unit test suite `tests/custom_objects/test_formula_engine.py` covering:
- Arithmetic (`(amount * probability) / 100`).
- Conditionals (`IF(stage == 'Closed Won', 'Success', 'In Progress')`).
- String manipulation (`CONCAT(first_name, ' ', last_name)`).
- Malicious AST rejection (`__import__`, `os.system`, file reading).
- Division by zero and safe null handling.
### Sprint 4.5B: Schema & Model Extensions
- Extend `CustomFieldDefinition` with `formula_expression`, `formula_return_type`, `rollup_*` columns.
- Generate and run Django migrations.
- Update `CustomFieldDefinitionSerializer` and schema validation endpoints (`/api/schema/validate-formula/`).
- Wire `pre_save` signals on standard models (`Account`, `Contact`, `Lead`, `Opportunity`, `Activity`) and `CustomRecord`.
### Sprint 4.5C: Roll-Up Summary Aggregation Subsystem
- Create `custom_objects/rollup_engine.py` with ORM aggregation (`Sum`, `Count`, `Min`, `Max`).
- Connect `post_save` and `post_delete` signals to trigger automatic parent rollup recalculations.
- Test parent rollup recalculation on opportunity creation/updates.
### Sprint 4.5D: Frontend UI Integration & Deployment
- Update `frontend/src/views/CustomObjectsView.vue`:
- Formula & Roll-up option in New Custom Field modal.
- Interactive formula expression builder and "Check Syntax" validation button.
- Distinctive badges and type formatting in the Fields & Relationships table.
- Verify full test suite passing (115+ tests).
- Roll out to Forgejo mirror and production Kubernetes cluster at `https://crm.ch4t.buzz`.
---
## 9. Verification & Acceptance Criteria
1. **Safety Assurance**: The expression parser must raise `ValidationError` against any unapproved AST nodes, builtins, or imports.
2. **Formula Execution**: Updating record fields automatically triggers formula recomputation on save; verified via API responses.
3. **Cycle Prevention**: Circular formula definitions (e.g. `A` references `B`, `B` references `A`) are rejected at schema creation time.
4. **Roll-Up Parity**: Creating or updating an Opportunity with `amount = 50000` on an Account recalculates the Account's `total_opportunity_amount__c` roll-up field immediately.
5. **UI Fidelity**: Object Manager displays formula and rollup fields with full metadata, formula syntax inspectability, and standard Lightning styling.