Connectors

Connectors are the foundation for data access in TraceMem. They establish and maintain connections to external data systems (databases, SaaS platforms) that Data Products use to read and write data.

What Connectors Are

A Connector represents:

• A named, configured connection to an external system
• Stored credentials (encrypted)
• Connection status and health monitoring
• Schema discovery and versioning
• A stable identity for governance and tracing

Connectors enable TraceMem to:

• Access data across heterogeneous systems
• Monitor connection health in real-time
• Track schema changes automatically
• Provide unified error handling
• Maintain secure credential storage

What Connectors Are Not

A Connector is not:

• A Data Product (connectors provide infrastructure, Data Products provide governance)
• A direct interface for agents (agents access data through Data Products)
• A workflow engine
• A data transformation layer
• A replacement for business logic

Connectors provide the "how" of data access. Data Products provide the "what" and "why".

Connector Types

Connectors come in different types based on the external system they connect to:

Database Connectors

Connect to relational databases and provide:

• Schema discovery (tables, columns, types, constraints)
• Read and write operations
• Transaction support
• Connection pooling
• Health monitoring

SaaS Connectors

Connect to SaaS platforms via APIs and provide:

• Object discovery (entities, fields, relationships)
• API-based read/write operations
• Authentication token management
• Rate limiting and error handling

Connector Lifecycle

1. Creation

Connectors are created by administrators via:

• The Admin Dashboard
• The Admin API

Agents cannot create or modify Connectors.

Connectors can be created using:

• Explicit configuration (connection details provided separately)
• Connection strings (URL-like format for simplified setup)

The exact format depends on the connector type and system being connected to.

2. Connection Status

Every connector has two status fields:

status - Administrative status

• enabled - Connector is active and can be used
• disabled - Connector is administratively disabled
• deprecated - Connector is marked for removal

connection_status - Real-time connection health

• connected - Last connection attempt succeeded
• disconnected - Not currently connected (initial state)
• error - Last connection attempt failed

3. Schema Discovery

For database connectors, the system automatically:

• Discovers schema structure (tables, columns, types)
• Generates a schema hash
• Detects schema changes
• Creates new schema versions only when structural changes occur

The schema watcher job:

• Runs periodically (default: every 5 minutes)
• Compares current schema with previous version
• Only creates a new version if HasChanges = true
• Triggers resource synchronization on changes

4. Testing and Validation

Connectors can be tested to verify connectivity and configuration. Testing:

• Attempts to establish a connection
• Updates connection status fields
• Returns connection health information
• Does not affect existing connections

5. Disabling and Deprecation

Disabling a connector:

• Prevents new data access operations
• Preserves configuration and credentials
• Can be re-enabled
• Existing decision traces remain valid

Deprecating a connector:

• Soft-delete (connector_id never reused)
• Marks for eventual cleanup
• Preserves historical references in traces

Connector Security

Credential Storage

Credentials are:

• Encrypted at rest using AES-256
• Never returned in API responses (only { "stored": true })
• Stored separately from configuration
• Cannot be rotated by TraceMem (external systems manage credentials)

Access Control

Connectors are:

• Tenant-scoped (isolated per tenant)
• Only accessible via Admin API (JWT authentication)
• Never directly accessible to agents
• Used indirectly through Data Products

Relationship to Other Concepts

Connectors → Data Products

Connectors and Data Products work together to provide governed data access:

1. Connectors provide the infrastructure:

   • Connection to external systems
   • Schema discovery and versioning
   • Read/write execution
   • Connection health monitoring

2. Data Products provide the governance:

   • Purpose-bound access
   • Exposed schema definition (subset of source schema)
   • Operation controls (read, insert, update, delete)
   • Policy enforcement
   • Restrictions and compliance rules

A Data Product references one or more Connectors in its sources. The relationship is:

• Connectors → how to access data (infrastructure layer)
• Data Products → what and why to access data (governance layer)

Agents never interact with Connectors directly. They access data through Data Products, which use Connectors under the hood.

Connectors → Decision Traces

Connectors contribute to decision traces through:

• Schema hashes in decision-time snapshots
• Connection metadata in trace events
• Historical validity (traces remain valid even if connectors are deprecated)

When a decision is made, the connector's schema version is captured in the snapshot, ensuring that "why this was allowed" remains answerable even if schemas change later.

Connection Management

Connection Affinity

In multi-node deployments, TraceMem manages connection affinity to ensure persistent connections remain on the same node. This is important because:

• Database connections are stateful
• SaaS API connections may maintain session state
• Connection pooling requires affinity

The system tracks which node owns each connection and routes requests accordingly.

Health Monitoring

Connectors continuously monitor their connection health:

• Connection status (connected, disconnected, error)
• Last connection attempt timestamp
• Last successful connection timestamp
• Error messages (sanitized for security)

This enables proactive monitoring and alerting when connections fail.

Where Connectors Sit in the Flow

External System (Database/SaaS)
    ↑
    | (via Connector)
    ↓
Data Product
    ↑
    | (via Agent MCP)
    ↓
Agent (in Decision Envelope)

Key points:

• Agents never see Connectors directly
• Data Products abstract Connectors from agents
• Connectors provide the infrastructure layer
• Data Products provide the governance layer

Best Practices

1. Use descriptive names - Indicate environment, system type, and purpose
2. Test connectivity - Verify connections after creation and credential updates
3. Monitor connection status - Regularly check connection health
4. Rotate credentials externally - Update connector configuration after credential rotation in external systems
5. One connector per system - Create separate connectors for different environments or systems
6. Secure credential storage - Never expose credentials in logs or error messages

Mental Model

Connectors are the plumbing.
Data Products are the faucets.
Agents turn on the faucets, but they never see the plumbing.
The plumbing knows how to get water from the source.
The faucets control how much water flows and for what purpose.
If the plumbing breaks, the faucets stop working, but agents only see that the faucet is broken, not why.