Docs Packages

`@procedures/s3`

packages/client-s3/README.md on GitHub · Package map · Procedures

AWS S3 operations as RPC procedures. Upload, download, list, delete, and perform multipart uploads via client.call().

Overview#

client-s3 provides comprehensive AWS S3 access through the client procedure ecosystem with:

Standard operations: Upload, download, list, delete objects
Multipart uploads: Large file uploads with resumable parts
Environment-based config: AWS credentials from environment variables
Zod validation: Runtime type safety on all inputs
MinIO/LocalStack support: Custom S3 endpoints for testing

The package auto-registers 9 procedures:

Standard: s3.upload, s3.download, s3.list, s3.listAll, s3.delete
Multipart: s3.multipart.init, s3.multipart.upload, s3.multipart.complete, s3.multipart.abort

Architecture#

Diagram source (Mermaid). GitHub draws it on the page of the file.

graph TB
    subgraph "Client Ecosystem"
        Client["@procedures/client"]
        Registry["Procedure Registry"]
    end

    subgraph "client-s3 Package"
        Register["register.ts<br/>(Auto-registers procedures)"]
        Types["types.ts<br/>(Zod Schemas + Types)"]
        S3Client["s3-client.ts<br/>(Singleton S3Client)"]

        subgraph "Procedure Categories"
            StandardOps["Standard Operations<br/>upload, download, list, delete"]
            MultipartOps["Multipart Operations<br/>init, upload, complete, abort"]
        end
    end

    subgraph "AWS SDK"
        AWSS3["@aws-sdk/client-s3"]
        Commands["PutObjectCommand<br/>GetObjectCommand<br/>ListObjectsV2Command<br/>DeleteObjectCommand<br/>CreateMultipartUploadCommand<br/>UploadPartCommand<br/>CompleteMultipartUploadCommand<br/>AbortMultipartUploadCommand"]
    end

    subgraph "Configuration"
        EnvVars["Environment Variables<br/>AWS_ACCESS_KEY_ID<br/>AWS_SECRET_ACCESS_KEY<br/>AWS_REGION<br/>S3_ENDPOINT"]
    end

    subgraph "S3 Storage"
        S3["AWS S3<br/>or<br/>MinIO/LocalStack"]
    end

    Register --> Client
    Register --> Registry

    StandardOps --> S3Client
    MultipartOps --> S3Client

    S3Client --> EnvVars
    S3Client --> AWSS3
    AWSS3 --> Commands
    Commands --> S3

    Types -.validates.-> StandardOps
    Types -.validates.-> MultipartOps

    style Register fill:#e1f5ff
    style S3Client fill:#fff4e1
    style Types fill:#f0f0f0

Package Dependency Flow#

Diagram source (Mermaid). GitHub draws it on the page of the file.

graph LR
    App["Your Application"]
    ClientS3["@procedures/s3"]
    Client["@procedures/client"]
    Zod["zod"]
    AWSSDK["@aws-sdk/client-s3"]

    App --> ClientS3
    ClientS3 --> Client
    ClientS3 --> Zod
    ClientS3 --> AWSSDK

    App -.client.call.-> Client
    Client -.executes.-> ClientS3
    ClientS3 -.validates.-> Zod
    ClientS3 -.S3 ops.-> AWSSDK

Installation#

npm install @procedures/s3

Peer Dependencies#

{
  "@procedures/client": "github:mark1russell7/client#main",
  "@aws-sdk/client-s3": "^3.705.0",
  "zod": "^3.24.0"
}

Configuration#

Environment Variables#

# Static credentials (optional — omit to use an IAM role / SSO / profile)
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_SESSION_TOKEN=your-session-token  # For STS/SSO temporary credentials

# Optional
AWS_REGION=us-east-1  # Default: us-east-1
S3_ENDPOINT=http://localhost:9000  # For MinIO/LocalStack

The package supplies explicit static credentials only when both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set (adding AWS_SESSION_TOKEN when present). If they are not set, it falls back to the AWS SDK default credential provider chain — SSO, IAM roles/instance profiles (IMDS), and shared config profiles all work with no code changes. Never hardcode credentials in your code.

S3 Client Configuration#

The package uses a singleton S3Client that's automatically configured from environment variables:

import { getS3Config, getS3Client, resetS3Client } from "@procedures/s3";

// Get current config (from environment)
const config = getS3Config();
console.log(config.region);

// Get singleton S3 client
const s3 = getS3Client();

// Reset client (useful for testing or credential refresh)
resetS3Client();

API Reference#

Standard Operations#

`s3.upload`#

Upload content to an S3 bucket.

const result = await client.exec(
  ["s3", "upload"],
  {
    bucket: "my-bucket",
    key: "files/document.pdf",
    body: fileContent,  // String or base64 encoded
    contentType: "application/pdf",
    base64: true,  // If body is base64 encoded
    metadata: {
      uploadedBy: "user123",
      timestamp: "2024-03-15T10:30:00Z"
    }
  }
);

console.log(result.etag);
console.log(result.location);  // s3://my-bucket/files/document.pdf

Input:

interface S3UploadInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key (path in bucket)
  body: string;                      // Content (string or base64)
  contentType?: string;              // MIME type
  base64?: boolean;                  // If true, body is base64 encoded
  metadata?: Record<string, string>; // Custom metadata
}

Output:

interface S3UploadOutput {
  etag: string;                      // Object ETag
  versionId?: string;                // Version ID (if versioning enabled)
  location: string;                  // Full S3 URI
}

`s3.download`#

Download content from an S3 bucket.

const result = await client.exec(
  ["s3", "download"],
  {
    bucket: "my-bucket",
    key: "files/document.pdf",
    encoding: "utf8",  // Omit for binary (returns base64)
    versionId: "abc123"  // Optional specific version
  }
);

console.log(result.body);  // Content as string or base64
console.log(result.contentType);
console.log(result.contentLength);

Input:

interface S3DownloadInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  encoding?: string;                 // Text encoding (utf8, ascii, etc.)
                                     // Omit for binary (returns base64)
  versionId?: string;                // Specific version to download
}

Output:

interface S3DownloadOutput {
  body: string;                      // Content (string if encoding specified, base64 otherwise)
  contentLength: number;             // Size in bytes
  contentType?: string;              // MIME type
  etag?: string;                     // Object ETag
  lastModified?: string;             // ISO timestamp
}

`s3.list`#

List objects in an S3 bucket.

const result = await client.exec(
  ["s3", "list"],
  {
    bucket: "my-bucket",
    prefix: "files/",
    maxKeys: 100,
    continuationToken: undefined,  // For pagination
    delimiter: "/"  // For hierarchical listing
  }
);

console.log(`Found ${result.keyCount} objects`);
result.contents.forEach(obj => {
  console.log(`${obj.key} - ${obj.size} bytes`);
});

if (result.isTruncated) {
  console.log("More results available with token:", result.nextContinuationToken);
}

Input:

interface S3ListInput {
  bucket: string;                    // S3 bucket name
  prefix?: string;                   // Filter by prefix
  maxKeys?: number;                  // Max objects to return (default: 1000)
  continuationToken?: string;        // Pagination token
  delimiter?: string;                // Delimiter for hierarchical listing (e.g., "/")
}

Output:

interface S3ListOutput {
  contents: S3ListObject[];          // List of objects
  commonPrefixes: string[];          // Common prefixes (when using delimiter)
  isTruncated: boolean;              // More results available
  nextContinuationToken?: string;    // Token for next page
  keyCount: number;                  // Number of keys returned
}

interface S3ListObject {
  key: string;                       // Object key
  size: number;                      // Size in bytes
  lastModified: string;              // ISO timestamp
  etag: string;                      // Object ETag
  storageClass?: string;             // Storage class (STANDARD, GLACIER, etc.)
}

`s3.listAll`#

List every object under a bucket/prefix, following continuationToken automatically. Use this when a bucket/prefix may contain more than 1000 keys — s3.list returns at most one page, whereas s3.listAll loops until the result is no longer truncated and returns the aggregated contents.

const result = await client.exec(
  ["s3", "listAll"],
  {
    bucket: "my-bucket",
    prefix: "snapshots/",
    maxKeys: 1000,   // Optional: page size per underlying request
    delimiter: "/"   // Optional: hierarchical listing
  }
);

console.log(`Found ${result.keyCount} objects across ${result.pageCount} pages`);

Input:

interface S3ListAllInput {
  bucket: string;                    // S3 bucket name
  prefix?: string;                   // Filter by prefix
  maxKeys?: number;                  // Page size per request (default/max 1000)
  delimiter?: string;                // Delimiter for hierarchical listing (e.g., "/")
}

Output:

interface S3ListAllOutput {
  contents: S3ListObject[];          // All objects (not capped at 1000)
  commonPrefixes: string[];          // Deduplicated common prefixes across pages
  keyCount: number;                  // Total number of keys returned
  pageCount: number;                 // Number of ListObjectsV2 requests made
}

Warning: s3.listAll buffers the full key set in memory and issues one request per 1000 keys. For very large buckets prefer paging with s3.list + continuationToken.

`s3.delete`#

Delete an object from an S3 bucket.

const result = await client.exec(
  ["s3", "delete"],
  {
    bucket: "my-bucket",
    key: "files/old-document.pdf",
    versionId: "abc123"  // Optional specific version
  }
);

console.log(result.deleted);  // true

Input:

interface S3DeleteInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  versionId?: string;                // Specific version to delete
}

Output:

interface S3DeleteOutput {
  deleted: boolean;                  // Whether deletion was successful
  deleteMarker?: boolean;            // Delete marker created (if versioning enabled)
  versionId?: string;                // Version ID of delete marker
}

Multipart Upload Operations#

For large files (>5MB), use multipart upload:

`s3.multipart.init`#

Initialize a multipart upload.

const result = await client.exec(
  ["s3", "multipart", "init"],
  {
    bucket: "my-bucket",
    key: "large-files/video.mp4",
    contentType: "video/mp4",
    metadata: {
      originalName: "vacation-video.mp4",
      uploadedBy: "user123"
    }
  }
);

console.log(result.uploadId);  // Save this for subsequent part uploads

Input:

interface S3MultipartInitInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  contentType?: string;              // MIME type
  metadata?: Record<string, string>; // Custom metadata
}

Output:

interface S3MultipartInitOutput {
  uploadId: string;                  // Upload ID for subsequent operations
  bucket: string;                    // Bucket name
  key: string;                       // Object key
}

`s3.multipart.upload`#

Upload a part in a multipart upload.

const result = await client.exec(
  ["s3", "multipart", "upload"],
  {
    bucket: "my-bucket",
    key: "large-files/video.mp4",
    uploadId: uploadId,  // From init
    partNumber: 1,  // Part number (1-10000)
    body: partData  // Base64 encoded part data
  }
);

console.log(result.etag);  // Save etag for complete
console.log(result.partNumber);

Input:

interface S3MultipartUploadInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  uploadId: string;                  // Upload ID from init
  partNumber: number;                // Part number (1-10000)
  body: string;                      // Part content (base64 encoded)
}

Output:

interface S3MultipartUploadOutput {
  etag: string;                      // Part ETag (required for complete)
  partNumber: number;                // Part number
}

`s3.multipart.complete`#

Complete a multipart upload by assembling parts.

const result = await client.exec(
  ["s3", "multipart", "complete"],
  {
    bucket: "my-bucket",
    key: "large-files/video.mp4",
    uploadId: uploadId,
    parts: [
      { partNumber: 1, etag: "etag1" },
      { partNumber: 2, etag: "etag2" },
      { partNumber: 3, etag: "etag3" }
    ]
  }
);

console.log(result.location);  // Final object location
console.log(result.etag);      // Final object ETag

Input:

interface S3MultipartCompleteInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  uploadId: string;                  // Upload ID
  parts: Array<{                     // Uploaded parts (ordered by partNumber)
    etag: string;
    partNumber: number;
  }>;
}

Output:

interface S3MultipartCompleteOutput {
  location: string;                  // Object location URL
  bucket: string;                    // Bucket name
  key: string;                       // Object key
  etag: string;                      // Final object ETag
}

`s3.multipart.abort`#

Abort a multipart upload and clean up parts.

const result = await client.exec(
  ["s3", "multipart", "abort"],
  {
    bucket: "my-bucket",
    key: "large-files/video.mp4",
    uploadId: uploadId
  }
);

console.log(result.aborted);  // true

Input:

interface S3MultipartAbortInput {
  bucket: string;                    // S3 bucket name
  key: string;                       // Object key
  uploadId: string;                  // Upload ID to abort
}

Output:

interface S3MultipartAbortOutput {
  aborted: boolean;                  // Whether abort was successful
}

Usage Examples#

Basic Upload and Download#

import { Client, LocalTransport, PROCEDURE_REGISTRY } from "@procedures/client";
import "@procedures/s3"; // registers its procedures

const client = new Client(new LocalTransport({ registry: PROCEDURE_REGISTRY }));

// Upload text content
await client.exec(
  ["s3", "upload"],
  {
    bucket: "my-bucket",
    key: "data/config.json",
    body: JSON.stringify({ env: "production" }),
    contentType: "application/json"
  }
);

// Upload binary file (base64 encoded)
const fileBuffer = await fs.promises.readFile("./image.png");
const base64Content = fileBuffer.toString("base64");

await client.exec(
  ["s3", "upload"],
  {
    bucket: "my-bucket",
    key: "images/photo.png",
    body: base64Content,
    base64: true,
    contentType: "image/png",
    metadata: {
      uploadedBy: "user123",
      originalName: "vacation-photo.png"
    }
  }
);

// Download text content
const { body } = await client.exec(
  ["s3", "download"],
  {
    bucket: "my-bucket",
    key: "data/config.json",
    encoding: "utf8"
  }
);

const config = JSON.parse(body);

// Download binary content (returns base64)
const { body: imageBase64 } = await client.exec(
  ["s3", "download"],
  {
    bucket: "my-bucket",
    key: "images/photo.png"
  }
);

const imageBuffer = Buffer.from(imageBase64, "base64");

List and Delete Objects#

// List all objects in a prefix
const { contents } = await client.exec(
  ["s3", "list"],
  {
    bucket: "my-bucket",
    prefix: "data/",
    maxKeys: 1000
  }
);

console.log(`Found ${contents.length} objects`);

// List with pagination
let continuationToken: string | undefined;
do {
  const result = await client.exec(
    ["s3", "list"],
    {
      bucket: "my-bucket",
      prefix: "logs/",
      maxKeys: 100,
      continuationToken
    }
  );

  result.contents.forEach(obj => {
    console.log(`${obj.key} - ${obj.size} bytes`);
  });

  continuationToken = result.nextContinuationToken;
} while (continuationToken);

// Delete objects
for (const obj of contents) {
  await client.exec(
    ["s3", "delete"],
    {
      bucket: "my-bucket",
      key: obj.key
    }
  );
}

Hierarchical Listing (Folders)#

// List "folders" in a bucket
const { contents, commonPrefixes } = await client.exec(
  ["s3", "list"],
  {
    bucket: "my-bucket",
    prefix: "data/",
    delimiter: "/"
  }
);

console.log("Files in data/:");
contents.forEach(obj => {
  console.log(`  ${obj.key}`);
});

console.log("Subdirectories:");
commonPrefixes.forEach(prefix => {
  console.log(`  ${prefix}`);
});

Multipart Upload (Large Files)#

const PART_SIZE = 5 * 1024 * 1024;  // 5MB parts

// 1. Initialize upload
const { uploadId } = await client.exec(
  ["s3", "multipart", "init"],
  {
    bucket: "my-bucket",
    key: "videos/large-file.mp4",
    contentType: "video/mp4"
  }
);

try {
  // 2. Upload parts
  const file = await fs.promises.readFile("./large-file.mp4");
  const parts = [];

  for (let i = 0; i < file.length; i += PART_SIZE) {
    const partNumber = Math.floor(i / PART_SIZE) + 1;
    const partData = file.slice(i, Math.min(i + PART_SIZE, file.length));
    const base64Part = partData.toString("base64");

    const { etag } = await client.exec(
      ["s3", "multipart", "upload"],
      {
        bucket: "my-bucket",
        key: "videos/large-file.mp4",
        uploadId,
        partNumber,
        body: base64Part
      }
    );

    parts.push({ partNumber, etag });
    console.log(`Uploaded part ${partNumber}`);
  }

  // 3. Complete upload
  const result = await client.exec(
    ["s3", "multipart", "complete"],
    {
      bucket: "my-bucket",
      key: "videos/large-file.mp4",
      uploadId,
      parts
    }
  );

  console.log(`Upload complete: ${result.location}`);

} catch (error) {
  // 4. Abort on error
  await client.exec(
    ["s3", "multipart", "abort"],
    {
      bucket: "my-bucket",
      key: "videos/large-file.mp4",
      uploadId
    }
  );
  throw error;
}

Backup Script Example#

import { glob } from "fast-glob";

// Backup directory to S3
const files = await glob("./data/**/*", { onlyFiles: true });

for (const file of files) {
  const content = await fs.promises.readFile(file);
  const s3Key = file.replace("./data/", "backups/");

  await client.exec(
    ["s3", "upload"],
    {
      bucket: "my-backup-bucket",
      key: s3Key,
      body: content.toString("base64"),
      base64: true,
      metadata: {
        backupDate: new Date().toISOString(),
        originalPath: file
      }
    }
  );

  console.log(`Backed up: ${file} -> s3://my-backup-bucket/${s3Key}`);
}

Using with MinIO or LocalStack#

# Set custom endpoint for local S3-compatible storage
export S3_ENDPOINT=http://localhost:9000
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1

// Works the same way with MinIO/LocalStack
await client.exec(
  ["s3", "upload"],
  {
    bucket: "test-bucket",
    key: "test.txt",
    body: "Hello MinIO!"
  }
);

Integration with Ecosystem#

Bundle Packages#

client-s3 can be included in bundle packages:

// In bundle-data/src/register.ts
import "@procedures/s3/register.js";
import "@procedures/mongo"; // registers its procedures
import "@procedures/fs"; // registers its procedures

Dependencies#

Runtime Dependencies#

@procedures/client - Core client procedure system
@aws-sdk/client-s3 (^3.705.0) - AWS SDK for S3
zod (^3.24.0) - Runtime type validation

Development Dependencies#

@mark1russell7/cue - Configuration and type utilities
typescript (^5.9.3) - TypeScript compiler
@types/node (^22.0.0) - Node.js type definitions

Package Structure#

client-s3/
├── src/
│   ├── types.ts                      # Zod schemas + TypeScript types
│   ├── s3-client.ts                  # S3Client singleton factory
│   ├── register.ts                   # Auto-registration
│   ├── index.ts                      # Main entry point
│   ├── procedures/
│   │   └── s3/
│   │       ├── index.ts              # Procedure exports
│   │       ├── upload.ts             # s3.upload
│   │       ├── download.ts           # s3.download
│   │       ├── list.ts               # s3.list
│   │       ├── delete.ts             # s3.delete
│   │       ├── multipart-init.ts     # s3.multipart.init
│   │       ├── multipart-upload.ts   # s3.multipart.upload
│   │       ├── multipart-complete.ts # s3.multipart.complete
│   │       └── multipart-abort.ts    # s3.multipart.abort
├── dist/                             # Compiled output
├── package.json
├── tsconfig.json
└── README.md

Development#

Build#

npm run build

Type Check#

npm run typecheck

Clean#

npm run clean

Security#

Never hardcode AWS credentials in code
Prefer IAM roles / instance profiles when running on AWS infrastructure: leave AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY unset and the SDK default credential provider chain resolves role credentials automatically
Use static AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (plus AWS_SESSION_TOKEN for temporary SSO/STS credentials) only where an IAM role is not available
Restrict S3 bucket policies to minimum required permissions
Enable S3 bucket versioning for important data
Consider S3 bucket encryption at rest

License#

MIT

Author#

Mark Russell marktheprogrammer17@gmail.com

Overview#

Architecture#

Package Dependency Flow#

Installation#

Peer Dependencies#

Configuration#

Environment Variables#

S3 Client Configuration#

API Reference#

Standard Operations#

s3.upload#

s3.download#

s3.list#

s3.listAll#

s3.delete#

Multipart Upload Operations#

s3.multipart.init#

s3.multipart.upload#

s3.multipart.complete#

s3.multipart.abort#

Usage Examples#

Basic Upload and Download#

List and Delete Objects#

Hierarchical Listing (Folders)#

Multipart Upload (Large Files)#

Backup Script Example#

Using with MinIO or LocalStack#

Integration with Ecosystem#

Bundle Packages#

Dependencies#

Runtime Dependencies#

Development Dependencies#

Package Structure#

Development#

Build#

Type Check#

Clean#

Security#

License#

Author#

`s3.upload`#

`s3.download`#

`s3.list`#

`s3.listAll`#

`s3.delete`#

`s3.multipart.init`#

`s3.multipart.upload`#

`s3.multipart.complete`#

`s3.multipart.abort`#