Storage provider abstraction — Google Drive, Dropbox, OneDrive, AWS S3, and SFTP implementations.

Storage Providers Architecture

The storage system provides a unified interface for storing encrypted documents across multiple backends.

Overview

┌───────────────────────────────────────────────────────────────────────┐
│                        Application Layer                              │
│                                                                       │
│                  Storage Manager (interface)                          │
│                        │                                              │
│        ┌───────────┬───┼───────┬────────────┬───────────┐            │
│        ▼           ▼   ▼       ▼            ▼           ▼            │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ ┌─────────┐     │
│  │ Google   │ │ Dropbox  │ │ OneDrive │ │   S3   │ │  SFTP   │     │
│  │  Drive   │ │          │ │          │ │        │ │         │     │
│  └──────────┘ └──────────┘ └──────────┘ └────────┘ └─────────┘     │
│        │           │           │           │          │               │
│        ▼           ▼           ▼           ▼          ▼               │
│   OAuth 2.0   OAuth 2.0   OAuth 2.0    AWS SDK    SSH/FTP           │
│   (Google)    (Dropbox)   (MS Graph)                                 │
└───────────────────────────────────────────────────────────────────────┘

Interface

Storage Interface

type StorageProvider interface {
    Type() StorageType
    
    // Connection
    Connect(ctx context.Context, config StorageConfig) error
    Disconnect(ctx context.Context) error
    Test(ctx context.Context) error
    
    // File operations
    Upload(ctx context.Context, path string, data io.Reader) error
    Download(ctx context.Context, path string) (io.ReadCloser, error)
    Delete(ctx context.Context, path string) error
    
    // Navigation
    List(ctx context.Context, path string) ([]FileInfo, error)
}

File Info

type FileInfo struct {
    Name         string
    Path         string
    IsDirectory  bool
    Size         int64
    Modified     time.Time
}

Supported Providers

Google Drive

Property	Value
Type	gdrive
Auth	OAuth 2.0
API	Google Drive API v3

Features:

OAuth 2.0 flow
Folder selection
Automatic token refresh
Large file support (up to 5TB)

Configuration:

{
  "type": "gdrive",
  "access_token": "...",
  "refresh_token": "...",
  "folder_id": "..."
}

Implementation Highlights:

Token refresh handled automatically
Files uploaded as application/octet-stream (prevents processing)
Opaque file names ({uuid}.enc)

Dropbox

Property	Value
Type	dropbox
Auth	OAuth 2.0
API	Dropbox API v2

Features:

OAuth 2.0 flow
Folder selection
Automatic token refresh
Large file support (up to 350GB)

Configuration:

{
  "type": "dropbox",
  "access_token": "...",
  "refresh_token": "...",
  "path_prefix": "/BAP/wills"
}

OneDrive

Property	Value
Type	onedrive
Auth	OAuth 2.0
API	Microsoft Graph API v1.0

Features:

OAuth 2.0 flow (Microsoft identity platform v2.0)
Folder selection
Automatic token refresh
Personal and Work/School accounts
Anonymous sharing links for WTP

Configuration:

{
  "type": "onedrive",
  "access_token": "...",
  "refresh_token": "...",
  "folder_id": "root"
}

AWS S3

Property	Value
Type	s3
Auth	Access keys
SDK	AWS SDK for Go

Features:

Simple credential-based auth
Any S3-compatible storage (AWS, MinIO, DO Spaces)
Bucket versioning support
Presigned URLs for access

Configuration:

{
  "type": "s3",
  "access_key_id": "AKIA...",
  "secret_access_key": "...",
  "bucket": "my-bap-wills",
  "region": "us-east-1"
}

S3-Compatible:

{
  "type": "s3",
  "endpoint": "https://nyc3.digitaloceanspaces.com",
  "access_key_id": "...",
  "secret_access_key": "...",
  "bucket": "my-wills",
  "region": "us-east-1",
  "path_style": true
}

SFTP

Property	Value
Type	sftp
Auth	Password or SSH key
Protocol	SFTP/SSH

Features:

SSH key authentication
Password authentication
Self-hosted option
Full control

Configuration (Password):

{
  "type": "sftp",
  "host": "sftp.example.com",
  "port": 22,
  "username": "bap",
  "password": "..."
}

Configuration (SSH Key):

{
  "type": "sftp",
  "host": "sftp.example.com",
  "port": 22,
  "username": "bap",
  "ssh_key": "-----BEGIN OPENSSH PRIVATE KEY-----\n..."
}

File Handling

Upload Process

Receive file data (already encrypted)
Generate storage path: /wills/<will_id>/<doc_id>.enc
Upload with appropriate metadata
Verify upload success
Return storage path

Download Process

Receive storage path
Download encrypted blob
Return reader for decryption
Close connection

Path Structure

/BAP/wills/
└── {will-uuid}/
    ├── {doc1-uuid}.enc
    ├── {doc2-uuid}.enc
    └── {doc3-uuid}.enc

Security

Encryption Before Upload

All files are encrypted before upload:

Server encrypts with DEK
Storage provider never sees plaintext
Opaque filenames prevent correlation

Credential Storage

All credentials encrypted with master key
OAuth tokens encrypted at rest
No plaintext credentials in database

Provider Security

Provider	Security Features
Google Drive	OAuth 2.0, TLS 1.3
Dropbox	OAuth 2.0, TLS 1.3
AWS S3	IAM policies, TLS 1.3
SFTP	SSH encryption

Error Handling

Common Errors

Error	Cause	Handling
Invalid credentials	Wrong keys/tokens	Mark disconnected
Token expired	OAuth token old	Auto-refresh or reconnect
Permission denied	Bad IAM/bucket policy	Prompt reconnect
Not found	File deleted externally	Return error
Storage full	Provider quota	Alert user

Retry Logic

const maxRetries = 3

for i := 0; i < maxRetries; i++ {
    err := provider.Upload(ctx, path, data)
    if err == nil {
        return nil
    }
    if !isRetryable(err) {
        return err
    }
    time.Sleep(time.Second * time.Duration(i+1))
}

Testing

Connection Test

Each provider implements a test method:

func (s *S3Provider) Test(ctx context.Context) error {
    // Attempt to list bucket
    _, err := s.client.ListObjectsV2(s.bucket).MaxKeys(1).Send(ctx)
    return err
}

Upload Test

Test uploads a small dummy file to verify write permissions.

Monitoring

Metrics

Upload/download success rate
Average transfer time
Error rates by provider
Storage usage

Logging

All operations logged:

File uploads/downloads
Errors with context
Performance metrics

Adding New Providers

Interface Implementation

type NewStorageProvider struct {
    config StorageConfig
}

func (s *NewStorageProvider) Type() StorageType {
    return "newstorage"
}

func (s *NewStorageProvider) Connect(ctx context.Context, config StorageConfig) error {
    // Implementation
}

func (s *NewStorageProvider) Upload(ctx context.Context, path string, data io.Reader) error {
    // Implementation
}

// ... other interface methods

Best Practices

For Production

Use OAuth for GDrive/Dropbox (automatic refresh)
Enable bucket versioning on S3 (optional)
Use SFTP with SSH keys (not passwords)
Monitor storage quotas

For Security

Always encrypt before upload
Use opaque filenames
Upload as application/octet-stream
Never log credentials

Next Steps

Deployment - Docker Compose — Running with storage
Deployment - Production Checklist — Production setup

Storage Providers Architecture

On this page