Sepia Documentation

📖 Complete API Documentation: https://ralsina.github.io/sepia/api/Sepia.html (opens in new tab)

Sepia is a file-system-based serialization library for Crystal that provides intelligent object persistence with automatic filesystem watching and backup capabilities.

What is Sepia?

Sepia allows you to save and load Crystal objects as files and directories on the filesystem, with automatic relationship tracking and change detection. It bridges the gap between in-memory objects and persistent storage without the complexity of traditional databases.

Key Features

🗂️ File System Storage

• Serializable Objects: Store objects as individual files
• Container Objects: Store complex objects as directories with nested structures
• Automatic Relationships: Handles object references with symlinks
• Canonical Storage: Objects stored in consistent ClassName/object_id structure

👀 File System Watching

• Real-time Monitoring: Detect external file changes automatically
• Multiple Backends: Support for both fswatch and Linux inotify
• Smart Filtering: Eliminates self-generated events to prevent unnecessary callbacks
• Thread-safe: Concurrent access with proper synchronization

💾 Backup Functionality

• Complete Backups: Create tar archives of object trees with all relationships
• Inspection Tools: List contents and verify backup integrity
• API Integration: Simple backup methods from Storage and Object classes
• Symlink Preservation: Maintains object relationships in backups

🔄 Generation Tracking

• Optimistic Concurrency: Prevents lost updates with generation numbers
• Conflict Detection: Identifies concurrent modifications
• Automatic Merging: Smart handling of object relationships

Why Use Sepia?

• Simple Setup: No database servers or migrations required
• Human-readable: Objects stored as plain files and directories
• Development-friendly: Easy to debug, backup, and version control
• Real-time Updates: Automatic detection of external file changes
• Backup Ready: Built-in backup and restore capabilities
• Crystal Native: Designed specifically for the Crystal ecosystem

Quick Example

require "sepia"

# Define a simple document
class Document < Sepia::Object
  include Sepia::Serializable

  property title : String
  property content : String

  def initialize(@title = "", @content = "")
  end

  def to_sepia : String
    "#{@title}\n#{@content}"
  end

  def self.from_sepia(sepia_string : String) : self
    lines = sepia_string.split('\n', 2)
    new(lines[0]? || "", lines[1]? || "")
  end
end

# Configure storage
Sepia::Storage.configure(:filesystem, {"path" => "./data"})

# Create and save a document
doc = Document.new("My First Document", "Hello, Sepia!")
doc.save  # Automatically generates sepia_id

# Load it back
loaded = Sepia::Storage.get(Document, doc.sepia_id)
puts loaded.title  # "My First Document"

Installation

Add this to your application's shard.yml:

dependencies:
  sepia:
    github: ralsina/sepia

Then run:

shards install

Documentation Structure

This documentation is organized into several sections:

• Getting Started: Introduction, installation, and basic concepts
• User Guide: Comprehensive usage examples and feature explanations
• API Reference: Detailed API documentation
• Advanced Topics: Performance, troubleshooting, and advanced usage
• Examples: Real-world usage patterns and examples

Requirements

• Crystal 1.16.3 or higher
• Optional: fswatch shard for cross-platform file system watching
• Optional: inotify.cr shard for Linux-native file system monitoring

Sepia makes file system persistence simple, reliable, and intelligent for Crystal applications.

Getting Started

Welcome to Sepia! This guide will help you get up and running with Sepia's file-system-based object persistence.

What You'll Learn

In this section, you'll learn:

• How to install Sepia in your Crystal project
• Basic concepts and terminology
• How to create your first persistent objects
• How to organize your data structures

Prerequisites

Before you begin, make sure you have:

• Crystal 1.16.3 or higher installed
• Basic understanding of Crystal classes and modules
• A code editor or IDE

Core Concepts

Sepia provides two main modules for object persistence:

Serializable Objects

Objects that serialize to a single file on disk. Perfect for simple data structures like documents, configurations, or settings.

Container Objects

Objects that serialize as directories containing other objects. Great for complex data structures with relationships like project hierarchies, user profiles, or configuration systems.

Storage Backends

Sepia supports multiple storage backends:

• Filesystem (default): Stores objects in local directories
• Memory: Keeps objects in RAM (useful for testing)

Generation Tracking

Sepia tracks object versions to prevent conflicts when multiple processes modify the same data.

Next Steps

1. Installation - Add Sepia to your project
2. Quick Start - Create your first persistent objects
3. Core Concepts - Understand the fundamentals

Let's start with Installation.

Installation

This guide will help you install Sepia in your Crystal project and set up the necessary dependencies.

Adding Sepia to Your Project

Using Shards

1. Add Sepia to your shard.yml file:

dependencies:
  sepia:
    github: ralsina/sepia

2. Run the installation command:

shards install

3. Require Sepia in your Crystal code:

require "sepia"

Optional Dependencies

Sepia includes optional dependencies for enhanced functionality:

File System Watching

For cross-platform file system monitoring:

dependencies:
  sepia:
    github: ralsina/sepia
  fswatch:
    github: bcardiff/crystal-fswatch

For Linux-native monitoring (recommended for Linux):

dependencies:
  sepia:
    github: ralsina/sepia
  inotify:
    github: petoem/inotify.cr

Static Builds

If you're building a static binary, you can compile without fswatch:

crystal build src/your_app.cr -D no_fswatch

Sepia will automatically use a no-op fallback for file watching when fswatch is not available.

Basic Configuration

After installation, configure Sepia to use your preferred storage backend:

require "sepia"

# Use the default filesystem backend
Sepia::Storage.configure(:filesystem, {"path" => "./data"})

# Or use in-memory storage (useful for testing)
Sepia::Storage.configure(:memory)

Verifying Installation

Create a simple test to verify Sepia is working:

require "sepia"

class TestObject < Sepia::Object
  include Sepia::Serializable

  property name : String

  def initialize(@name = "")
  end

  def to_sepia : String
    @name
  end

  def self.from_sepia(sepia_string : String) : self
    new(sepia_string)
  end
end

# Configure storage
Sepia::Storage.configure(:filesystem, {"path" => "./test_data"})

# Create and save an object
obj = TestObject.new("Hello Sepia!")
obj.save

puts "Sepia is working! Object saved with ID: #{obj.sepia_id}"

Run this with:

crystal run test_installation.cr

If successful, you should see output similar to:

Sepia is working! Object saved with ID: test-object-123e4567-e89b-12d3-a456-426614174000

Next Steps

Now that Sepia is installed, let's move on to the Quick Start guide to create your first persistent objects!

Quick Start

This guide will get you up and running with Sepia in just a few minutes. We'll create a simple document management system to demonstrate the core concepts.

Step 1: Define Your Objects

Let's create a simple Document class that can be saved and loaded:

require "sepia"

class Document < Sepia::Object
  include Sepia::Serializable

  property title : String
  property content : String

  def initialize(@title = "", @content = "")
  end

  def to_sepia : String
    "#{@title}\n#{@content}"
  end

  def self.from_sepia(sepia_string : String) : self
    lines = sepia_string.split('\n', 2)
    new(lines[0]? || "", lines[1]? || "")
  end
end

Step 2: Configure Storage

# Configure Sepia to use the filesystem backend
Sepia::Storage.configure(:filesystem, {"path" => "./data"})

Step 3: Save and Load Objects

# Create a new document
doc = Document.new("My First Document", "Hello, Sepia!")
doc.save  # Automatically generates a unique ID

puts "Document saved with ID: #{doc.sepia_id}"

# Load it back
loaded_doc = Sepia::Storage.get(Document, doc.sepia_id).as(Document)
puts "Loaded title: #{loaded_doc.title}"
puts "Loaded content: #{loaded_doc.content}"

Step 4: Try Container Objects

Container objects can contain other objects. Let's create a Folder:

class Folder < Sepia::Object
  include Sepia::Container

  property name : String
  property description : String?
  property documents : Array(Document)

  def initialize(@name = "", @description = nil)
    @documents = [] of Document
  end
end

# Create a folder with documents
folder = Folder.new("Important Docs", "My important documents")

doc1 = Document.new("Meeting Notes", "Discussed project timeline")
doc1.sepia_id = "meeting-notes"
doc1.save

doc2 = Document.new("Todo List", "- Review code\n- Write docs")
doc2.sepia_id = "todo-list"
doc2.save

folder.documents << doc1 << doc2
folder.sepia_id = "important-docs"
folder.save

puts "Folder saved with #{folder.documents.size} documents"

Step 5: File System Watching

Sepia can detect when files are changed externally:

# Set up a watcher
storage = Sepia::Storage.backend.as(Sepia::FileStorage)
watcher = Sepia::Watcher.new(storage)

watcher.on_change do |event|
  puts "Detected change: #{event.type} - #{event.object_class}:#{event.object_id}"

  # Reload the object if it was modified
  if event.type.modified?
    begin
      obj = Sepia::Storage.load(event.object_class.constantize(typeof(Object)), event.object_id)
      puts "Reloaded: #{obj}"
    rescue ex
      puts "Failed to reload: #{ex.message}"
    end
  end
end

# Start watching
watcher.start

# Now try editing one of the document files manually:
# ./data/Document/meeting-notes
# You should see the watcher detect the change!

# Stop watching when done
# watcher.stop

Step 6: Backup Your Data

Sepia makes it easy to create backups:

# Backup a specific object
backup_path = doc.backup_to("my_document_backup.tar")
puts "Backup created: #{backup_path}"

# Backup multiple objects
backup_path = Sepia::Storage.backup([doc1, doc2, folder], "project_backup.tar")
puts "Project backup created: #{backup_path}"

# Inspect backup contents
manifest = Sepia::Backup.list_contents("project_backup.tar")
puts "Backup contains #{manifest.all_objects.values.map(&.size).sum} objects"

# Verify backup integrity
result = Sepia::Backup.verify("project_backup.tar")
puts "Backup is #{result.valid ? "valid" : "invalid"}"

Complete Example

Here's a complete working example:

require "sepia"

class Document < Sepia::Object
  include Sepia::Serializable

  property title : String
  property content : String

  def initialize(@title = "", @content = "")
  end

  def to_sepia : String
    "#{@title}\n#{@content}"
  end

  def self.from_sepia(sepia_string : String) : self
    lines = sepia_string.split('\n', 2)
    new(lines[0]? || "", lines[1]? || "")
  end
end

# Configure storage
Sepia::Storage.configure(:filesystem, {"path" => "./quickstart_data"})

# Create and save a document
doc = Document.new("Welcome to Sepia", "This is your first persistent object!")
doc.save

puts "✅ Document saved!"
puts "   Title: #{doc.title}"
puts "   ID: #{doc.sepia_id}"

# Load it back
loaded = Sepia::Storage.get(Document, doc.sepia_id).as(Document)
puts "✅ Document loaded!"
puts "   Title: #{loaded.title}"

# Create a backup
backup_path = doc.backup_to("welcome_backup.tar")
puts "✅ Backup created: #{backup_path}"

puts "\n🎉 Quick start completed!"

Save this as quickstart.cr and run:

crystal run quickstart.cr

What's Next?

Now that you have the basics, explore these topics:

• Serializable Objects - Deep dive into file-based objects
• Container Objects - Learn about complex object structures
• File System Watching - Build reactive applications
• Backup and Restore - Data protection strategies

Happy coding with Sepia! 🚀

Core Concepts

User Guide

Serializable Objects

Container Objects

Storage Management

Generation Tracking

Generation tracking enables optimistic concurrency control and versioning of objects. This is particularly useful for collaborative applications where multiple users might edit the same data.

Overview

Generation tracking allows you to:

• Track multiple versions of the same object
• Detect concurrent modifications
• Implement optimistic locking
• Maintain a complete history of changes
• Recover previous versions when needed

How It Works

ID Format

Objects are stored with IDs in the format: {base_id}.{generation}

• base_id: The unique identifier (typically a UUID)
• generation: Version number starting from 0

Example files on disk:

data/
  └── Note/
      ├── note-123e4567-e89b-12d3-a456-426614174000.0
      ├── note-123e4567-e89b-12d3-a456-426614174000.1
      └── note-123e4567-e89b-12d3-a456-426614174000.2

Atomic Operations

Each save_with_generation creates a new file, ensuring:

• No data corruption during writes
• Previous versions remain intact
• Easy rollback to any version

API Reference

Instance Methods

generation : Int32

Returns the generation number extracted from the object's ID.

note = Note.load("note-xxx.2")
note.generation  # => 2

base_id : String

Returns the base ID without the generation suffix.

note = Note.load("note-xxx.2")
note.base_id  # => "note-xxx"

save_with_generation : self

Creates a new version with incremented generation number.

note = Note.load("note-xxx.1")
new_note = note.save_with_generation
new_note.sepia_id  # => "note-xxx.2"

stale?(expected_generation : Int32) : Bool

Checks if a newer version exists.

note = Note.load("note-xxx.1")
note.stale?(1)  # Returns true if note-xxx.2 exists

Class Methods

latest(base_id : String) : self?

Returns the newest version of an object.

latest = Note.latest("note-xxx")
latest.generation  # Highest generation number

versions(base_id : String) : Array(self)

Returns all versions sorted by generation.

versions = Note.versions("note-xxx")
versions.map(&.generation)  # => [0, 1, 2, ...]

exists?(id : String) : Bool

Checks if an object with the given ID exists.

Note.exists?("note-xxx.1")  # => true
Note.exists?("note-xxx.99") # => false

Use Cases

Collaborative Editing

class Document < Sepia::Object
  include Sepia::Serializable

  property content : String
  property last_modified : Time

  # ... implement to_sepia/from_sepia
end

# When a user opens a document
doc = Document.load("doc-123.3")
user_generation = doc.generation

# When saving
if doc.stale?(user_generation)
  # Show merge dialog
  latest = Document.latest(doc.base_id)
  # Merge content
else
  doc.save_with_generation
end

Version History

# Show document history
versions = Document.versions("doc-123")
versions.each do |version|
  puts "Version #{version.generation}: #{version.last_modified}"
end

# Restore specific version
doc = Document.load("doc-123.2")
current = doc.save_with_generation  # Creates version 3 from version 2

Audit Trail

class LogEntry < Sepia::Object
  include Sepia::Serializable

  property action : String
  property user_id : String
  property timestamp : Time
  property data : String

  # ... serialization methods
end

# Each log change creates a new version
entry = LogEntry.new("update", "user1", Time.now, data_json)
entry.save_with_generation

# Query all changes
all_changes = LogEntry.versions("log-entry-456")

Performance Considerations

Storage Usage

• Each version consumes additional disk space
• Consider cleanup strategies for old versions

Lookup Performance

• latest() scans all files to find highest generation
• Consider caching for frequently accessed objects

Cleanup Strategies

# Keep only last N versions
def keep_recent_versions(base_id, max_versions = 10)
  versions = MyClass.versions(base_id)
  if versions.size > max_versions
    versions[0..-(max_versions+1)].each(&:delete)
  end
end

# Keep versions newer than date
def keep_recent_by_date(base_id, cutoff_date = Time.now - 30.days)
  versions = MyClass.versions(base_id)
  versions.select { |v| v.saved_at < cutoff_date }.each(&:delete)
end

Best Practices

1. Always check for staleness before saving in collaborative scenarios
2. Use meaningful base IDs that reflect the object's identity
3. Implement cleanup for long-running applications
4. Handle merge conflicts gracefully with user-friendly UI
5. Consider using GenerationInfo module to include generation data in JSON

Migration from Non-Generation IDs

Existing objects without generation suffix automatically work:

• Treated as generation 0
• Can be versioned normally
• No migration required

# Old code still works
old_obj = MyClass.load("old-id")
old_obj.generation  # => 0

# Start versioning
new_obj = old_obj.save_with_generation
new_obj.generation  # => 1

File System Watching

Sepia's file system watcher enables reactive applications that respond to external changes in storage. It's particularly useful for collaborative applications where multiple users or processes might modify data simultaneously.

Backend Options

The file watcher supports multiple backends for cross-platform compatibility:

fswatch (Default)

• Platforms: Linux, macOS, Windows
• Requirements: libfswatch installed
• Use case: Development and cross-platform applications

inotify

• Platforms: Linux only
• Requirements: None (uses kernel inotify)
• Use case: Production Linux servers, static builds

Configuration

The file watcher is automatically configured based on the backend you selected when building your application.

# No additional configuration needed - uses the compiled backend
storage = Sepia::Storage.backend.as(Sepia::FileStorage)
watcher = Sepia::Watcher.new(storage)

Event Logging in Sepia

Sepia provides a comprehensive event logging system that tracks object lifecycle events and user activities. This is particularly useful for collaborative applications, audit trails, and activity feeds.

Table of Contents

• Overview
• Key Concepts
• Enabling Event Logging
• Automatic Event Logging
• Activity Logging
• Querying Events
• Event Structure
• On-Disk Format
• Generation Tracking
• Storage API
• Object API
• Use Cases
• Best Practices
• Advanced Topics

Overview

Sepia's event logging system captures all important actions on objects:

• Lifecycle Events: Created, Updated, Deleted operations
• Activity Events: User-defined activities that aren't object persistence
• Metadata Support: Rich JSON-serializable context for events
• Generation Tracking: Links events to object versions for optimistic concurrency
• Per-Object Storage: Events stored alongside object data for easy access

Key Concepts

Event Types

• Created: Object was created for the first time
• Updated: Object was modified and saved
• Deleted: Object was removed from storage
• Activity: User-defined action (e.g., moved_lane, highlighted, shared)

Metadata

All events support arbitrary JSON-serializable metadata:

# Simple metadata
{"user" => "alice", "reason" => "initial_creation"}

# Complex metadata
{
  "user" => "alice",
  "timestamp" => Time.utc,
  "changes" => ["title", "content"],
  "collaboration" => {
    "session_id" => "abc123",
    "duration" => 1800,
    "participants" => ["alice", "bob"]
  }
}

Generation Tracking

Events are linked to object generation numbers:

• Generation 0: Activity events (don't change object state)
• Generation N+1: Save operations that create new versions
• Current generation: Delete operations

Enabling Event Logging

Event logging is disabled by default. Enable it per class:

class Document < Sepia::Object
  include Sepia::Serializable
  sepia_log_events true  # Enable logging for this class

  property content : String

  def initialize(@content = "")
  end

  def to_sepia : String
    @content
  end

  def self.from_sepia(json : String) : self
    new(json)
  end
end

class Project < Sepia::Object
  include Sepia::Container
  sepia_log_events true  # Enable logging for containers too

  property name : String
  property boards : Array(Board)

  def initialize(@name = "")
    @boards = [] of Board
  end
end

Configuration Options

# Enable logging
sepia_log_events true

# Disable logging
sepia_log_events false

# Alternative syntax
sepia_log_events_enabled
sepia_log_events_disabled

Automatic Event Logging

When event logging is enabled, Sepia automatically logs lifecycle events:

Using Storage API

# Create and save (logs Created event)
doc = Document.new("Hello World")
doc.sepia_id = "my-doc"
Sepia::Storage.save(doc, metadata: {"user" => "alice", "reason" => "initial_creation"})

# Update and save (logs Updated event)
doc.content = "Updated content"
Sepia::Storage.save(doc, metadata: {"user" => "bob", "reason" => "content_edit"})

# Delete (logs Deleted event)
Sepia::Storage.delete(doc, metadata: {"user" => "admin", "reason" => "cleanup"})

Using Object API

# Create and save
doc = Document.new("Hello World")
doc.sepia_id = "my-doc"
doc.save(metadata: {"user" => "alice"})  # Smart save (auto-detects new vs existing)

# Update with forced generation
doc.content = "Updated content"
doc.save(force_new_generation: true, metadata: {"user" => "bob"})

# Or use the legacy method
new_doc = doc.save_with_generation(metadata: {"user" => "charlie"})

Activity Logging

Log user activities that aren't related to object persistence:

Basic Activity Logging

# Log activities on any object
doc.log_activity("highlighted", {"color" => "yellow", "user" => "alice"})
doc.log_activity("shared", {"platform" => "slack", "user" => "bob"})

# Log activities on containers too
project.log_activity("lane_created", {"lane_name" => "Review", "user" => "charlie"})
project.log_activity("color_changed")  # Simple version without metadata

Rich Activity Examples

# Complex activity with structured metadata
note.log_activity("moved", {
  "from_lane" => "In Progress",
  "to_lane" => "Done",
  "user" => "alice",
  "timestamp" => Time.utc,
  "drag_duration" => 2.5,
  "collaborators" => ["bob", "charlie"]
})

# Activity with arrays and objects
board.log_activity("restructured", {
  "action" => "lane_reorder",
  "user" => "alice",
  "changes" => [
    {"lane" => "Todo", "old_index" => 0, "new_index" => 1},
    {"lane" => "Done", "old_index" => 2, "new_index" => 0}
  ],
  "affected_items" => 5
})

Querying Events

Access the event history for any object:

Basic Querying

# Get all events for an object
events = Sepia::Storage.object_events(Document, "my-doc")

# Events are ordered by timestamp (newest first)
events.each do |event|
  puts "#{event.timestamp}: #{event.event_type}"
  puts "  Generation: #{event.generation}"
  puts "  Metadata: #{event.metadata}"
end

Filtering Events

# Filter by event type
created_events = events.select(&.event_type.created?)
updated_events = events.select(&.event_type.updated?)
activity_events = events.select(&.event_type.activity?)
deleted_events = events.select(&.event_type.deleted?)

# Filter by generation
gen2_events = events.select(&.generation.==(2))

# Filter by metadata
user_events = events.select { |e| e.metadata["user"]? == "alice" }

Advanced Queries

# Get activities by specific user
alice_activities = events.select do |event|
  event.event_type.activity? &&
  event.metadata["user"]? == "alice"
end

# Get recent save operations
recent_saves = events.select do |event|
  event.event_type.created? || event.event_type.updated?
end.first(5)

# Get activities in time range
today_events = events.select do |event|
  event.timestamp > Time.utc.at_beginning_of_day
end

Event Structure

Each event contains rich information:

event = Sepia::LogEvent.new(
  event_type: Sepia::LogEventType::Updated,
  generation: 2,
  metadata: {"user" => "alice", "reason" => "edit"}
)

event.timestamp    # => Time when the event occurred
event.event_type   # => Created, Updated, Deleted, or Activity
event.generation   # => Object generation number (0 for activities)
event.metadata     # => JSON::Any with custom context

Accessing Metadata

# Simple metadata access
user = event.metadata["user"].as_s
reason = event.metadata["reason"].as_s

# Complex metadata access
changes = event.metadata["changes"].as_a
collaborators = event.metadata["collaborators"].as_a.map(&.as_s)

# Safe access
user = event.metadata["user"]?.try(&.as_s)
count = event.metadata["count"]?.try(&.as_i)

On-Disk Format

Events are stored in JSON Lines format in .events/ directories:

Directory Structure

./_data
├── Document
│   └── doc-123
│       └── .events
│           └── doc-123.jsonl    # One JSON event per line
└── Project
    └── proj-456
        └── .events
            └── proj-456.jsonl

Event File Format

Each line is a JSON object:

{"ts":"2025-01-15T10:30:45Z","type":"created","gen":1,"meta":{"user":"alice","reason":"initial_creation"}}
{"ts":"2025-01-15T10:31:20Z","type":"activity","gen":1,"meta":{"action":"highlighted","color":"yellow","user":"bob"}}
{"ts":"2025-01-15T10:32:10Z","type":"updated","gen":2,"meta":{"user":"charlie","reason":"content_edit"}}
{"ts":"2025-01-15T10:33:00Z","type":"deleted","gen":2,"meta":{"user":"admin","reason":"cleanup"}}

Field Descriptions

• ts: Timestamp in RFC3339 format
• type: Event type (created, updated, deleted, activity)
• gen: Generation number (0 for activities, N+1 for saves, current for deletes)
• meta: JSON metadata object (preserves original data types)

Generation Tracking

Events are properly linked to object generation numbers:

Generation Logic

# Object at generation 0 (new)
doc.save()                    # Creates generation 1, logs Created(gen=1)

# Object at generation 1 (existing)
doc.save()                    # Creates generation 2, logs Updated(gen=2)
doc.log_activity("highlighted")  # Logs Activity(gen=1) - uses current generation
doc.save(force_new_generation: true)  # Creates generation 3, logs Updated(gen=3)

# Delete always uses current generation
doc.delete()                  # Logs Deleted(gen=3)

Timeline Example

gen=1 (Created)    ← Initial save
gen=1 (Activity)   ← Activity on generation 1
gen=2 (Updated)    ← Save creates generation 2
gen=2 (Activity)   ← Activity on generation 2
gen=2 (Deleted)    ← Delete uses current generation

Storage API

Basic Operations

# Save with smart detection
Sepia::Storage.save(object, metadata: {"user" => "alice"})

# Save with forced new generation
Sepia::Storage.save(object, force_new_generation: true, metadata: {"user" => "bob"})

# Save to custom location
Sepia::Storage.save(object, path: "/custom/path", metadata: {"user" => "charlie"})

# Delete with metadata
Sepia::Storage.delete(object, metadata: {"user" => "admin", "reason" => "cleanup"})

Advanced Options

# Disable caching
Sepia::Storage.save(object, cache: false, metadata: {"user" => "alice"})

# Custom path with metadata
Sepia::Storage.save(object,
  path: "/archive/documents",
  cache: false,
  metadata: {"user" => "archiver", "auto_archived" => true}
)

# Force new generation with complex metadata
Sepia::Storage.save(object,
  force_new_generation: true,
  metadata: {
    "user" => "alice",
    "batch_id" => "batch_123",
    "changes" => {
      "fields_modified" => ["title", "content"],
      "word_count_change" => 150
    }
  }
)

Object API

Simple Operations

# Smart save (auto-detects new vs existing)
doc.save()

# Save with metadata
doc.save(metadata: {"user" => "alice"})

# Force new generation
doc.save(force_new_generation: true)

# Force new generation with metadata
doc.save(force_new_generation: true, metadata: {"user" => "bob"})

Activity Logging

# Simple activity
doc.log_activity("highlighted")

# Activity with metadata
doc.log_activity("shared", {"platform" => "slack", "user" => "charlie"})

# Complex activity with rich metadata
doc.log_activity("collaborative_edit", {
  "session_duration" => 1800,
  "participants" => ["alice", "bob", "charlie"],
  "changes_made" => 15,
  "conflicts_resolved" => 2
})

Method Chaining

# All save methods return self for chaining
doc.save(metadata: {"user" => "alice"})
  .log_activity("processed", {"processor" => "auto-summarizer"})
  .save(force_new_generation: true, metadata: {"user" => "system"})

Use Cases

Activity Feeds

# Get recent activity for display
recent_activities = board_events
  .select(&.event_type.activity?)
  .first(10)
  .map do |event|
    {
      "action" => event.metadata["action"],
      "user" => event.metadata["user"],
      "timestamp" => event.timestamp,
      "details" => event.metadata
    }
  end

Audit Trails

# Complete change history for compliance
audit_trail = Sepia::Storage.object_events(Document, doc_id)
  .map do |event|
    {
      "timestamp" => event.timestamp,
      "action" => event.event_type.to_s,
      "user" => event.metadata["user"]?,
      "reason" => event.metadata["reason"]?,
      "generation" => event.generation
    }
  end

User Analytics

# User activity patterns
user_activities = all_events
  .select { |e| e.metadata["user"]? == "alice" }
  .group_by(&.timestamp.date)
  .transform_values(&.size)

puts "Alice's activity by day:"
user_activities.each do |date, count|
  puts "#{date}: #{count} actions"
end

Debugging

# Understand object lifecycle
puts "Document lifecycle:"
Sepia::Storage.object_events(Document, doc_id).each do |event|
  case event.event_type
  when .created?
    puts "  Created at #{event.timestamp} by #{event.metadata["user"]}"
  when .updated?
    puts "  Updated to gen#{event.generation} at #{event.timestamp} by #{event.metadata["user"]}"
  when .activity?
    puts "  Activity: #{event.metadata["action"]} at #{event.timestamp}"
  when .deleted?
    puts "  Deleted at #{event.timestamp} by #{event.metadata["user"]}"
  end
end

Best Practices

1. Enable Logging Judiciously

# ✅ DO: Enable for important user-facing objects
class Document < Sepia::Object
  include Sepia::Serializable
  sepia_log_events true  # Users care about document history
end

# ❌ DON'T: Enable for everything
class CacheEntry < Sepia::Object
  include Sepia::Serializable
  sepia_log_events false  # Internal objects don't need logging
end

2. Use Structured Metadata

# ✅ GOOD: Structured, queryable metadata
doc.log_activity("approved", {
  "approver" => "alice",
  "approval_level" => "manager",
  "criteria_met" => ["content_review", "fact_check"],
  "next_review" => 7.days.from_now
})

# ❌ AVOID: Unstructured strings
doc.log_activity("approved", "alice approved it as manager, content and facts checked, review in a week")

3. Choose Meaningful Actions

# ✅ GOOD: Specific, actionable events
note.log_activity("moved_to_lane", {"lane" => "Done"})
note.log_activity("assigned", {"assignee" => "bob"})
note.log_activity("priority_changed", {"old" => "high", "new" => "urgent"})

# ❌ AVOID: Generic or unclear events
note.log_activity("action", {"what" => "something happened"})

4. Include Context

# ✅ GOOD: Rich context for understanding
note.log_activity("edited", {
  "user" => "alice",
  "editor" => "web",
  "session_duration" => 120,
  "chars_added" => 150,
  "chars_removed" => 25,
  "auto_save" => false
})

# ❌ AVOID: Minimal context
note.log_activity("edited", {"user" => "alice"})

5. Use Generation Tracking

# ✅ DO: Use generation-aware operations when versioning matters
version = document.save_with_generation(metadata: {"user" => "alice"})
conflict_resolution = handle_conflicts_if_needed(version)

# ✅ DO: Use smart save for simple updates
document.save(metadata: {"user" => "bob"})

Advanced Topics

Custom Event Types

While Sepia provides built-in event types, you can extend the system:

# Add custom action metadata for specialized workflows
case study.log_activity("phase_change", {
  "action" => "phase_change",
  "from_phase" => "recruitment",
  "to_phase" => "interview",
  "protocol" => "clinical_trial_v2"
})

Event Filtering

# Create specialized queries
module EventQueries
  def self.user_activity(user_id : String, object_class : Class, limit = 50)
    Sepia::Storage.object_events(object_class, "*")
      .select { |e| e.metadata["user"]?.try(&.to_s) == user_id }
      .first(limit)
  end

  def self.recent_activities(hours = 24)
    cutoff = Time.utc - hours.hours
    # Implementation would need to scan multiple object files
    # Consider maintaining a global activity index for performance
  end
end

Performance Considerations

• Event Storage: Events are stored as JSON Lines for efficient append-only operations
• Query Performance: Reading events requires file I/O, consider caching frequent queries
• Storage Size: Events are stored alongside objects, monitor disk usage for high-frequency logging
• Indexing: For large-scale applications, consider maintaining separate indexes for common queries

Migration and Compatibility

Event storage format may change between Sepia versions. When upgrading:

1. Backup existing event data
2. Test with sample data
3. Run migration tools if provided
4. Verify event integrity

Current event format is stable but consider future enhancements:

• Event compression for large metadata
• Global activity indexes
• Event retention policies
• Cross-object event relationships

API Reference

Core Classes

• Sepia::LogEvent: Individual event record
• Sepia::LogEventType: Event type enum (Created, Updated, Deleted, Activity)
• Sepia::EventLogger: Event storage and retrieval engine

Key Methods

Storage API

Sepia::Storage.save(object, metadata?, cache?, path?, force_new_generation?)
Sepia::Storage.delete(object, metadata?, cache?)
Sepia::Storage.object_events(class, id) -> Array(LogEvent)

Object API

object.save(metadata?)
object.save(*, force_new_generation : Bool, metadata?)
object.save_with_generation(metadata?)
object.log_activity(action, metadata?)

Event Access

event.timestamp      # Time
event.event_type     # LogEventType
event.generation     # Int32
event.metadata       # JSON::Any

For more detailed API documentation, see the Crystal API docs.

Backup and Restore

Sepia provides comprehensive backup functionality that allows you to create tar archives of object trees, inspect backup contents, and verify backup integrity.

Overview

Sepia's backup system enables you to:

• Create complete backups of object trees with all relationships
• Inspect backup contents without restoring
• Verify backup integrity and structure
• Preserve symlinks and object relationships
• Generate human-readable metadata

Basic Usage

Creating Backups

Backup Multiple Objects

# Backup a list of root objects
root_objects = [document1, project1, user_profile]
backup_path = Sepia::Backup.create(root_objects, "project_backup.tar")
puts "Backup created: #{backup_path}"

Backup from Storage

# Backup specific objects from storage
objects = [
  Sepia::Storage.get(Document, "doc1"),
  Sepia::Storage.get(Project, "proj1")
]
backup_path = Sepia::Storage.backup(objects, "selected_backup.tar")

# Backup a single object
backup_path = Sepia::Storage.backup(document, "single_doc_backup.tar")

# Backup all objects in storage
backup_path = Sepia::Storage.backup_all("complete_backup.tar")

Backup from Individual Objects

# Simple backup to specific path
document.backup_to("doc_backup.tar")

# Backup with auto-generated filename (includes timestamp)
backup_path = document.create_backup()  # e.g., "document_abc123_20251125.tar"

# Backup to custom directory
document.backup_to("backups/docs/doc_backup.tar")

Inspecting Backups

List Backup Contents

manifest = Sepia::Backup.list_contents("project_backup.tar")

puts "Backup created at: #{manifest.created_at}"
puts "Version: #{manifest.version}"
puts "Root objects: #{manifest.root_objects.size}"

manifest.root_objects.each do |root_obj|
  puts "  - #{root_obj.class_name}/#{root_obj.object_id} (#{root_obj.object_type})"
end

puts "All objects by class:"
manifest.all_objects.each do |class_name, objects|
  puts "  - #{class_name}: #{objects.size} objects"
end

Get Backup Metadata

metadata = Sepia::Backup.get_metadata("project_backup.tar")
# Returns the same manifest as list_contents()

Verify Backup Integrity

result = Sepia::Backup.verify("project_backup.tar")

puts "Backup is #{result.valid ? "valid" : "invalid"}"
puts "Total objects: #{result.statistics.total_objects}"
puts "Serializable objects: #{result.statistics.serializable_objects}"
puts "Container objects: #{result.statistics.container_objects}"
puts "Object classes: #{result.statistics.classes.size}"

if result.errors.empty?
  puts "No verification errors"
else
  puts "Verification errors:"
  result.errors.each { |error| puts "  - #{error}" }
end

Backup Archive Structure

Sepia backups are standard tar archives with this structure:

backup.sepia.tar
├── metadata.json     # Backup manifest with object information
├── README           # Human-readable backup information
└── objects/         # All objects organized by class and ID
    ├── ClassName/object_id        # Serializable objects (files)
    └── ClassName/object_id/       # Container objects (directories)
        ├── data.json
        └── references/

metadata.json Format

{
  "version": "1.0",
  "created_at": "2025-11-25T17:30:00Z",
  "root_objects": [
    {
      "class_name": "Document",
      "object_id": "doc-123",
      "relative_path": "objects/Document/doc-123",
      "object_type": "Serializable"
    }
  ],
  "all_objects": {
    "Document": [
      {
        "class_name": "Document",
        "object_id": "doc-123",
        "relative_path": "objects/Document/doc-123",
        "object_type": "Serializable"
      }
    ],
    "Project": [...]
  }
}

Configuration

Sepia supports simple configuration for backup creation:

config = Sepia::Backup::Configuration.new

# Configure symlink handling
config.follow_symlinks = false  # Default: preserves symlinks as-is

# Create backup with configuration
backup_path = Sepia::Backup.create(root_objects, "backup.tar", config)

Configuration Presets

# Fast backup (no compression, minimal verification)
fast_config = Sepia::Backup::Configuration.fast

# Minimal backup (basic functionality only)
minimal_config = Sepia::Backup::Configuration.minimal

# Archive backup (maximum preservation)
archive_config = Sepia::Backup::Configuration.archive

backup_path = Sepia::Backup.create(root_objects, "archive.tar", archive_config)

Use Cases

Application Backup

class BackupService
  def backup_user_data(user : User)
    # Backup all objects related to a user
    root_objects = [user] + user.documents + user.projects

    timestamp = Time.utc.to_s("%Y%m%d_%H%M%S")
    backup_path = "backups/user_#{user.id}_#{timestamp}.tar"

    backup_path = Sepia::Backup.create(root_objects, backup_path)

    # Verify backup was successful
    result = Sepia::Backup.verify(backup_path)
    unless result.valid
      raise "Backup verification failed"
    end

    backup_path
  end
end

Project Export

def export_project(project : Project)
  # Create backup of entire project tree
  root_objects = [project] + find_all_related_objects(project)

  export_path = "exports/#{project.name}_export.tar"
  backup_path = Sepia::Backup.create(root_objects, export_path)

  puts "Project exported to: #{backup_path}"

  # Show what's in the export
  manifest = Sepia::Backup.list_contents(backup_path)
  puts "Export contains #{manifest.all_objects.values.map(&.size).sum} objects"
end

Data Migration

def migrate_data(source_storage, target_storage)
  # 1. Backup current data
  all_objects = source_storage.list_all_objects
  backup_path = Sepia::Backup.create(all_objects, "migration_backup.tar")

  # 2. Transfer data to new storage
  all_objects.each do |obj|
    target_storage.save(obj)
  end

  # 3. Verify migration
  result = Sepia::Backup.verify(backup_path)
  puts "Migration completed, backup verified: #{result.valid ? "✓" : "✗"}"
end

Scheduled Backups

class ScheduledBackup
  def initialize(@backup_dir : String)
    Dir.mkdir_p(@backup_dir)
  end

  def daily_backup
    timestamp = Time.utc.to_s("%Y%m%d")
    backup_path = File.join(@backup_dir, "daily_#{timestamp}.tar")

    # Backup all objects
    backup_path = Sepia::Storage.backup_all(backup_path)

    # Keep only last 30 daily backups
    cleanup_old_backups("daily_", 30)

    backup_path
  end

  private def cleanup_old_backups(prefix : String, keep_count : Int32)
    backups = Dir.glob(File.join(@backup_dir, "#{prefix}*.tar"))
      .sort_by { |f| File.basename(f) }
      .reverse

    if backups.size > keep_count
      backups[keep_count..-1].each do |old_backup|
        File.delete(old_backup)
        puts "Deleted old backup: #{old_backup}"
      end
    end
  end
end

Error Handling

Sepia provides specific exception types for backup operations:

begin
  backup_path = Sepia::Backup.create(root_objects, "backup.tar")
rescue Sepia::BackendNotSupportedError
  puts "Backup not supported with current storage backend"
rescue Sepia::BackupCreationError
  puts "Failed to create backup: check permissions and disk space"
rescue Sepia::BackupCorruptionError
  puts "Backup file is corrupted or invalid"
rescue ex : Exception
  puts "Unexpected error: #{ex.message}"
end

Performance Considerations

Large Backups

For large object trees:

• Consider filtering objects to exclude unnecessary data
• Use streaming for very large backups
• Monitor disk space availability
• Consider compression for network transfers

Backup Frequency

# Smart backup based on changes since last backup
class SmartBackup
  def backup_if_changed(last_backup_time : Time)
    modified_objects = find_objects_modified_since(last_backup_time)

    if modified_objects.empty?
      puts "No changes since last backup"
      return nil
    end

    backup_path = create_backup(modified_objects)
    puts "Backup created with #{modified_objects.size} modified objects"
    backup_path
  end

  private def find_objects_modified_since(time : Time)
    # Implementation depends on your storage backend
    # Could use file modification times or event logs
  end
end

Integration with External Tools

rsync Integration

backup_path = Sepia::Storage.backup_all("local_backup.tar")

# Sync to remote server
system("rsync", "-av", "local_backup.tar", "user@server:/backups/")

Cloud Storage

# After creating backup, upload to cloud service
backup_path = create_backup(objects)

# Upload using system tools
if File.exists?(backup_path)
  system("aws", "s3", "cp", backup_path, "s3://my-bucket/backups/")
  system("rclone", "copy", backup_path, "remote:backups/")
end

Best Practices

1. Always verify backups after creation
2. Regular testing of restore procedures
3. Monitor disk space for backup directories
4. Use meaningful filenames with timestamps
5. Document backup procedures for your team
6. Test backup restoration in staging environments
7. Consider encryption for sensitive data backups
8. Implement cleanup strategies for old backups

Restore Strategy

While Sepia focuses on backup creation (restore is application-specific), here's a general pattern:

class RestoreService
  def restore_from_backup(backup_path : String, target_storage)
    # 1. Inspect backup contents
    manifest = Sepia::Backup.list_contents(backup_path)

    # 2. Extract backup (application-specific logic)
    extract_and_process_backup(backup_path, target_storage, manifest)

    # 3. Rebuild relationships
    rebuild_object_relationships(manifest, target_storage)
  end

  # Implementation depends on your specific application requirements
  private def extract_and_process_backup(backup_path, storage, manifest)
    # Extract tar archive and process files according to your needs
  end
end

API Reference

Core Modules

Storage Backends

Utilities

Advanced Topics

Garbage Collection

Performance Considerations

Troubleshooting

Examples

Document Management

Configuration Management

Collaborative Applications