openforce/docs/FORMULA_FIELDS_SPEC.md
Bumble 4e64bcf3b4
Some checks failed
Build, Test, and Deploy / build-and-deploy (push) Failing after 29s
docs: add Salesforce Formula Fields and Roll-Up Summaries architectural specification
2026-09-27 04:42:39 +00:00

17 KiB

Salesforce Formula Fields & Roll-Up Summaries Architecture Specification

1. Executive Summary & Objective

In Salesforce, Formula Fields (Formula(Type)) and Roll-Up Summary Fields (RollupSummary) are core declaratively-defined computed metadata types. They enable administrators and developers to calculate values automatically, concatenate strings, evaluate conditional business logic, and aggregate child relationship data without writing imperative Apex code or triggers.

This specification details the architecture for implementing dynamic computed formula fields (formula) and roll-up summary aggregations (rollup_summary) within OpenForce across:

  1. Dynamic JSONB Custom Fields (__c) on both Standard Entities (Account, Contact, Lead, Opportunity, Activity, Campaign) and Custom Objects (CustomObject / CustomRecord).
  2. Standard Static Entity Models via serializer computation and database-level virtual properties.
  3. The AST-based Safe Expression Evaluator ensuring sandboxed, deterministic runtime computation with zero security risk (forbidding eval(), exec(), or arbitrary code execution).
  4. Dependency Graph Resolution & Dependency Ordering ensuring nested formulas (e.g. formula_b depending on formula_a) evaluate without stale data or infinite loops.
  5. The Unified Object Manager UI/UX allowing users to choose "Formula" and "Roll-Up Summary" field types, specify return types (Text, Number, Currency, Percent, Date, Checkbox), and test expressions in real time.

2. Salesforce Formula & Roll-Up Benchmark Analysis

2.1 Salesforce Formula Fields vs OpenForce Equivalents

Capability Salesforce Specification OpenForce Implementation
Storage Model Virtual / calculated at read-time (non-stored), or stored on save Computed on pre_save signal / serializer presentation; persisted in custom_fields JSONB for high-speed indexing & query capability
Return Types Checkbox, Currency, Date, DateTime, Number, Percent, Text boolean, currency, date, datetime, number, percent, text
Expression Syntax Excel-like functional syntax: IF(stage == 'Closed Won', amount * 0.1, 0) Excel/Salesforce compatible formula grammar compiled via Python ast / token parser
Cross-Object Lookups Spanning relations: Account.Industry, Contact.Account.Name Dot-notation traversal across Django foreign keys and custom lookup relationships
Standard Operators +, -, *, /, ^, ==, !=, <, >, <=, >=, &&, ||, & (concat) Full operator support with numeric promotion, safe null coercion, and string concatenation
Built-in Functions IF, CASE, ISBLANK, NOT, AND, OR, TEXT, VALUE, TODAY, NOW, CONCAT, UPPER, LOWER Built-in safe standard library module in custom_objects/formula_engine.py

2.2 Salesforce Roll-Up Summaries

Salesforce Roll-Up Summaries calculate values from a set of related detail records (Master-Detail relationships):

  • COUNT: Total number of child records.
  • SUM: Sum of a specific numeric/currency field across children.
  • MIN / MAX: Lowest or highest date/number across children.
  • Filter Criteria: Optional criteria (e.g. SUM(Opportunity.Amount) where Stage == 'Closed Won').

In OpenForce:

  • Supports reverse relations on standard models (e.g. Account.opportunities, Account.contacts) and CustomRecord child relations.
  • Evaluated via Django ORM aggregation (Sum, Count, Min, Max) triggered on child post_save and post_delete signals, atomically updating the parent's custom_fields['<field_name>'].

3. System Architecture & Component Design

+-------------------------------------------------------------------------------+
|                           Unified Object Manager (Vue 3)                      |
|  - Formula Field Creator (Return Type, Expression Editor, Syntax Check)       |
|  - Roll-Up Summary Creator (Target Child Object, Aggregation Type, Filter)    |
+---------------------------------------+---------------------------------------+
                                        | POST /api/custom-fields/
                                        v
+-------------------------------------------------------------------------------+
|                       Django REST Framework Schema & Validation               |
|  - Validates formula AST syntax against allowed tokens & referenced fields    |
|  - Cycle detection in formula dependency graphs (DAG)                          |
+---------------------------------------+---------------------------------------+
                                        |
                 +----------------------+-----------------------+
                 |                                              |
                 v                                              v
+------------------------------------+   +------------------------------------+
|    Formula Evaluation Engine       |   |    Roll-Up Aggregation Engine      |
|    (`formula_engine.py`)           |   |    (`rollup_engine.py`)            |
|  - Python AST Safe Visitor         |   |  - Django ORM Signal Triggers      |
|  - Field Resolver (Local + Lookup) |   |  - Parent Cache Invalidation       |
|  - Type Coercion & Formatters      |   |  - Count, Sum, Min, Max Execution  |
+-----------------+------------------+   +-----------------+------------------+
                  |                                        |
                  +-------------------+--------------------+
                                      |
                                      v
+-------------------------------------------------------------------------------+
|                      PostgreSQL Storage (`custom_fields` JSONB)               |
|  - Stored computed values indexed via PostgreSQL GIN indexes                   |
|  - Readily accessible in list views, filters, and workflow automations        |
+-------------------------------------------------------------------------------+

4. AST-Based Safe Expression Evaluation Engine

4.1 Security Constraints (Strict Sandboxing)

Under no circumstances should Python's native eval() or exec() be used. All expressions are parsed into an Abstract Syntax Tree (ast.parse(expr, mode='eval')) and walked using a strict ast.NodeVisitor:

  1. Forbidden Nodes: Call to unapproved functions, Import, ImportFrom, Lambda, ListComp, DictComp, GeneratorExp, Attribute (unless whitelisted for object traversal), Subscript, Assign, Delete, While, For.
  2. Allowed Nodes: BinOp, UnaryOp, Compare, Name, Constant, Call (only against registered whitelist), Attribute (for safe relation traversal e.g. account.name).

4.2 Supported Functions Whitelist

SAFE_FUNCTIONS = {
    # Logical
    'IF': lambda cond, true_val, false_val: true_val if cond else false_val,
    'AND': lambda *args: all(args),
    'OR': lambda *args: any(args),
    'NOT': lambda x: not bool(x),
    'CASE': case_evaluator,
    'ISBLANK': lambda val: val is None or val == '' or val == [],
    
    # Text / Formatting
    'CONCAT': lambda *args: ''.join(str(a) if a is not None else '' for a in args),
    'UPPER': lambda s: str(s or '').upper(),
    'LOWER': lambda s: str(s or '').lower(),
    'LEN': lambda s: len(str(s or '')),
    'TRIM': lambda s: str(s or '').strip(),
    'TEXT': lambda v: str(v if v is not None else ''),
    'VALUE': lambda s: float(s) if s not in (None, '') else 0.0,

    # Math
    'ROUND': lambda val, digits=0: round(float(val or 0), int(digits)),
    'ABS': lambda val: abs(float(val or 0)),
    'MOD': lambda a, b: (float(a) % float(b)) if float(b) != 0 else 0,
    'MAX': lambda *args: max(args) if args else None,
    'MIN': lambda *args: min(args) if args else None,
    'CEILING': lambda val: math.ceil(float(val or 0)),
    'FLOOR': lambda val: math.floor(float(val or 0)),

    # Date / Time
    'TODAY': lambda: timezone.now().date().isoformat(),
    'NOW': lambda: timezone.now().isoformat(),
    'YEAR': lambda d: parse_date(d).year if d else None,
    'MONTH': lambda d: parse_date(d).month if d else None,
    'DAY': lambda d: parse_date(d).day if d else None,
}

4.3 Reference Syntax & Field Resolution

Expressions can reference:

  1. Local Standard Fields: amount, stage, probability, first_name, annual_revenue.
  2. Local Custom Fields: commission_rate__c, discount_percentage__c.
  3. Cross-Object Relations (Dot notation):
    • account.name
    • account.annual_revenue
    • contact.account.industry

5. Data Model Schema Updates

5.1 CustomFieldDefinition Field Extensions

To support formulas and rollups, custom_objects/models.py extends CustomFieldDefinition:

class CustomFieldDefinition(models.Model):
    # Existing fields: entity_name, custom_object, name, label, field_type, is_required...

    FIELD_TYPE_CHOICES = [
        ('text', 'Text'),
        ('number', 'Number'),
        ('date', 'Date'),
        ('datetime', 'Date/Time'),
        ('boolean', 'Checkbox'),
        ('currency', 'Currency'),
        ('percent', 'Percent'),
        ('picklist', 'Picklist'),
        ('lookup', 'Lookup'),
        ('formula', 'Formula'),             # <--- New
        ('rollup_summary', 'Roll-Up Summary'), # <--- New
    ]

    # Formula configuration
    formula_expression = models.TextField(blank=True, default='')
    formula_return_type = models.CharField(
        max_length=30,
        choices=[
            ('text', 'Text'),
            ('number', 'Number'),
            ('currency', 'Currency'),
            ('percent', 'Percent'),
            ('date', 'Date'),
            ('datetime', 'Date/Time'),
            ('boolean', 'Checkbox'),
        ],
        blank=True,
        null=True
    )
    formula_decimal_places = models.IntegerField(default=2, blank=True)

    # Roll-up summary configuration
    rollup_target_entity = models.CharField(max_length=100, blank=True, null=True)
    rollup_relationship_field = models.CharField(max_length=100, blank=True, null=True)
    rollup_aggregate_type = models.CharField(
        max_length=20,
        choices=[
            ('COUNT', 'COUNT'),
            ('SUM', 'SUM'),
            ('MIN', 'MIN'),
            ('MAX', 'MAX'),
        ],
        blank=True,
        null=True
    )
    rollup_aggregate_field = models.CharField(max_length=100, blank=True, null=True)
    rollup_filter_criteria = models.JSONField(default=list, blank=True)

6. Execution Lifecycle & Dependency Graph

6.1 Dependency Ordering & Cycle Detection

When multiple formula fields exist on an entity (e.g., net_margin__c depends on gross_profit__c), we resolve dependencies using topological sorting:

Record Save / Evaluation Trigger
               │
               ▼
Extract all formula fields for Entity
               │
               ▼
Build Directed Graph of Field References
               │
               ▼
Topological Sort (Detect Cycles -> Raise ValidationError)
               │
               ▼
Evaluate in Topological Order:
  Step 1: Evaluate Independent Formulas
  Step 2: Inject evaluated values into context
  Step 3: Evaluate Dependent Formulas
               │
               ▼
Commit final calculated values to `custom_fields` JSONB

6.2 Pre-Save Signal Hook (crm_core/signals.py & custom_objects/signals.py)

A unified pre_save signal receiver intercepts record mutations:

@receiver(pre_save)
def evaluate_entity_formula_fields(sender, instance, **kwargs):
    entity_name = sender.__name__
    if not is_monitored_entity(entity_name):
        return
    
    # 1. Fetch all formula definitions for this entity
    formula_defs = get_active_formula_fields(entity_name)
    if not formula_defs:
        return

    # 2. Build record evaluation context (standard attributes + custom_fields)
    context = build_entity_context(instance)

    # 3. Evaluate formulas in topological order
    computed_values = evaluate_formulas_topological(formula_defs, context, instance)

    # 4. Merge back into custom_fields dictionary
    if not instance.custom_fields:
        instance.custom_fields = {}
    instance.custom_fields.update(computed_values)

6.3 Roll-Up Summary Signal Hook (post_save & post_delete)

When child records (e.g. Opportunity) are created, updated, or deleted, the parent entity's roll-up summary is recomputed:

@receiver(post_save, sender=Opportunity)
@receiver(post_delete, sender=Opportunity)
def trigger_opportunity_rollup(sender, instance, **kwargs):
    if instance.account_id:
        recalculate_rollups_for_record('Account', instance.account_id)

7. Frontend Specification: Object Manager Formula & Rollup Experience

7.1 Field Creation Wizard Flow (CustomObjectsView.vue)

  1. Step 1: Select Field Type
    • User selects Formula or Roll-Up Summary in the New Field modal.
  2. Step 2: Configure Return Type (for Formula)
    • User chooses: Text, Number, Currency, Percent, Date, Checkbox.
  3. Step 3: Expression Editor
    • Interactive syntax-highlighted textarea with auto-complete pills:
      • Insert Field: Dropdown of all standard and custom fields on the current entity and lookup parents.
      • Insert Operator: +, -, *, /, ==, !=, <, >, &&, ||.
      • Insert Function: IF(), CASE(), ISBLANK(), CONCAT(), ROUND(), TODAY().
    • "Check Syntax" button: Triggers POST /api/schema/validate-formula/ to check AST correctness and surface line-specific error messages before saving.
  4. Step 4: Configure Roll-Up Summary (for Roll-Up)
    • Target Child Object (e.g. Opportunities).
    • Roll-Up Type (COUNT, SUM, MIN, MAX).
    • Field to Aggregate (e.g. amount).
    • Filter criteria (e.g. stage == 'Closed Won').

7.2 Unified Table Representation

In the Fields & Relationships grid:

  • Origin Badge: Custom (Formula) or Custom (Roll-Up) styled with distinctive indigo and cyan badges.
  • Data Type Column: Shows Formula (Currency), Formula (Percent), or Roll-Up Summary (SUM).
  • Expression Preview: Clicking the row displays the read-only formula expression in an inspection drawer.

8. Sprint Roadmap: Implementation Phases

Sprint 4.5A: Core Safe AST Formula Engine & Test Suite

  • Create custom_objects/formula_engine.py:
    • AST parser, safe node visitor, token validator.
    • Whitelist of functions (IF, CASE, ISBLANK, CONCAT, ROUND, etc.).
    • Cross-object dot notation resolver.
    • Topological sorting & cycle detection for chained formulas.
  • Unit test suite tests/custom_objects/test_formula_engine.py covering:
    • Arithmetic ((amount * probability) / 100).
    • Conditionals (IF(stage == 'Closed Won', 'Success', 'In Progress')).
    • String manipulation (CONCAT(first_name, ' ', last_name)).
    • Malicious AST rejection (__import__, os.system, file reading).
    • Division by zero and safe null handling.

Sprint 4.5B: Schema & Model Extensions

  • Extend CustomFieldDefinition with formula_expression, formula_return_type, rollup_* columns.
  • Generate and run Django migrations.
  • Update CustomFieldDefinitionSerializer and schema validation endpoints (/api/schema/validate-formula/).
  • Wire pre_save signals on standard models (Account, Contact, Lead, Opportunity, Activity) and CustomRecord.

Sprint 4.5C: Roll-Up Summary Aggregation Subsystem

  • Create custom_objects/rollup_engine.py with ORM aggregation (Sum, Count, Min, Max).
  • Connect post_save and post_delete signals to trigger automatic parent rollup recalculations.
  • Test parent rollup recalculation on opportunity creation/updates.

Sprint 4.5D: Frontend UI Integration & Deployment

  • Update frontend/src/views/CustomObjectsView.vue:
    • Formula & Roll-up option in New Custom Field modal.
    • Interactive formula expression builder and "Check Syntax" validation button.
    • Distinctive badges and type formatting in the Fields & Relationships table.
  • Verify full test suite passing (115+ tests).
  • Roll out to Forgejo mirror and production Kubernetes cluster at https://crm.ch4t.buzz.

9. Verification & Acceptance Criteria

  1. Safety Assurance: The expression parser must raise ValidationError against any unapproved AST nodes, builtins, or imports.
  2. Formula Execution: Updating record fields automatically triggers formula recomputation on save; verified via API responses.
  3. Cycle Prevention: Circular formula definitions (e.g. A references B, B references A) are rejected at schema creation time.
  4. Roll-Up Parity: Creating or updating an Opportunity with amount = 50000 on an Account recalculates the Account's total_opportunity_amount__c roll-up field immediately.
  5. UI Fidelity: Object Manager displays formula and rollup fields with full metadata, formula syntax inspectability, and standard Lightning styling.