sky-buckets

sky-buckets is a TypeScript-powered Node.js package designed to streamline the process of uploading files from an Express backend using Multer directly to MinIO object storage. This package efficiently redirects file data streams, eliminating the need for temporary disk storage on your server, thereby reducing disk I/O and enhancing overall performance.

sky-buckets Overview

In modern web application development, handling file uploads often involves temporarily storing files on the server's local disk before moving them to a persistent storage solution like MinIO. This process can lead to significant disk I/O overhead and complexity in managing temporary files.

sky-buckets addresses these challenges by providing a custom Multer storage engine that streams incoming file data directly to MinIO. This means files never touch your server's disk, making it a highly efficient and scalable solution for applications dealing with high volumes of file uploads.

Key Features

• Direct Upload Redirection: Streams files directly from Multer to MinIO, bypassing temporary disk storage. [1, 2, 3]
• Full TypeScript Support: Built with TypeScript for type safety, maintainability, and an improved development experience. [4, 5]
• Configurable Multer Middleware: Seamless integration with Multer, allowing dynamic determination of MinIO bucket and object names based on request data. [6, 7, 8]
• Comprehensive Inline File Operations: Provides a MinioService for programmatic file operations outside the HTTP middleware context, including:
  • Upload from Buffer [1, 9]
  • Upload from Path [10, 11, 12]
  • Upload from Base64 [13, 14]
  • Upload from Blob [15]
  • Multi-file Upload [16]
  • Download from Specific Bucket [17, 10, 11]
  • Download File to Local Path [17, 10, 11]
  • Delete single file
  • Delete multiple file

Usage Documentation

Installation

To install the sky-buckets package and all its necessary dependencies, run the following command:

npm install sky-buckets

If you are using TypeScript in your project and directly importing types from express or multer outside of the sky-buckets package, you might also need to install the following type definitions as devDependencies in your project:

npm install --save-dev @types/node @types/express @types/multer

Note: The minio library versions 7.1.0 and above ship with built-in type definitions, so @types/minio is no longer separately required. [17, 10, 11, 18, 19]

Configuration

First, initialize your MinIO client. It is highly recommended to use environment variables for your MinIO credentials in production environments. [20, 11, 7]

import * as Minio from 'minio';

const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});

Usage as Multer Middleware

sky-buckets provides a custom Multer storage engine that redirects file uploads directly to MinIO. You can dynamically configure the bucket and object names based on the request. [6, 21, 22]

import express, { Request, Response, NextFunction } from 'express';
import multer from 'multer';
import * as Minio from 'minio';
import { createMinioStorage } from 'sky-buckets';

const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// MinIO Client Configuration (as above)
const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});

// Initialize Multer with the custom MinIO storage engine
const uploadToMinio = multer({
  storage: createMinioStorage({
    minioClient: minioClient,
    minioEndPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
    minioPort: parseInt(process.env.MINIO_PORT ?? '9000', 10),
    minioUseSSL: process.env.MINIO_USE_SSL === 'true',
    bucketName: (req) => req.body.bucketName ?? 'default-bucket',
    objectName: (req, file) => {
      const filename = req.body.filename ?? file.originalname;
      const timestamp = Date.now();
      return `${timestamp}-${filename}`;
    },
    metadata: (req, file) => ({
      'Content-Type': file.mimetype,
      'X-Uploaded-By': req.body.userId ?? 'anonymous',
    }),
  }),
});

// Example route for single file upload
// Use `uploadToMinio.single('fileFieldName')`
app.post('/upload/single', uploadToMinio.single('file'), (req: Request, res: Response) => {
  if (!req.file) {
    return res.status(400).json({ message: 'No file uploaded.' });
  }
  res.status(200).json({
    message: 'File uploaded successfully to MinIO!',
    fileInfo: req.file, // req.file will contain MinIO metadata like bucket, key, etag, location [23, 24]
  });
});

// Example route for multiple file uploads
// Use `uploadToMinio.array('fileFieldName', maxFileCount)`
app.post('/upload/multiple', uploadToMinio.array('files', 10), (req: Request, res: Response) => {
  if (!req.files |
| (Array.isArray(req.files) && req.files.length === 0)) {
    return res.status(400).json({ message: 'No files uploaded.' });
  }
  res.status(200).json({
    message: 'Files uploaded successfully to MinIO!',
    fileInfo: req.files, // req.files will contain an array of MinIO metadata [23, 24]
  });
});

// Multer error handling middleware
app.use((err: any, req: Request, res: Response, next: NextFunction) => {
  if (err instanceof multer.MulterError) {
    return res.status(400).json({
      code: err.code,
      message: err.message,
      field: err.field,
    });
  } else if (err) {
    console.error(err);
    return res.status(500).json({
      message: 'An unexpected error occurred.',
      error: err.message,
    });
  }
  next();
});

// Multer populates `req.body` with text fields and `req.file`/`req.files` with file metadata. [6, 7, 26]
// The custom storage engine can access `req.body` for `bucketName` and `filename` parsed by Multer. [6, 7, 8]

Inline File Operations with MinioService

sky-buckets also exports MinioService for direct interaction with MinIO, enabling programmatic upload and download operations outside the Multer middleware context.

import * as Minio from 'minio';
import { MinioService } from 'sky-buckets';

const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});

const minioService = new MinioService(minioClient);

// Upload from buffer
const buffer = Buffer.from(base64String, 'base64');
await minioService.uploadFromBuffer(
  'my-bucket',
  'example.png',
  buffer,
  { 'Content-Type': 'image/png' }
);

// Upload from filepath
await minioService.uploadFromPath(
  'my-bucket',
  'example.jpg',
  '/path/to/file.jpg',
  { 'Content-Type': 'image/jpeg' }
);

// Upload from base64 string
await minioService.uploadFromBase64(
  'my-bucket',
  'example.txt',
  base64String,
  { 'Content-Type': 'text/plain' }
);

// Upload from blob
const blob = new Blob([yourBufferOrString], { type: 'application/pdf' });
await minioService.uploadFromBlob(
  'my-bucket',
  'document.pdf',
  blob,
  { 'Content-Type': 'application/pdf' }
);

// Multifile upload
const files = [
  {
    objectName: 'file1.txt',
    data: Buffer.from('Hello 1'),
    metaData: { 'Content-Type': 'text/plain' }
  },
  {
    objectName: 'file2.txt',
    data: Buffer.from('Hello 2'),
    metaData: { 'Content-Type': 'text/plain' }
  }
];

await minioService.uploadMultipleFiles('my-bucket', files);

// Download stream to HTTP response
await minioService.downloadFile('my-bucket', 'example.png', res);

// Download to local path
await minioService.downloadFileToPath(
  'my-bucket',
  'example.png',
  './downloads/example.png'
);

// Delete single file 
await minioService.deleteFile('my-bucket', 'example.png');

// Delete multiple file
const filesToDelete = [
  'document1.pdf',
  'document2.docx',
  'spreadsheet.xlsx'
];

await minioService.deleteMultipleFiles('my-bucket', filesToDelete);

// Check if file exsist
const check = await minioService.fileExists('my-bucket', 'example.ong'); // Promise<boolean>

Error Handling

The package integrates error handling for both Multer and MinIO. Multer errors can be caught using instanceof multer.MulterError within an Express error handling middleware. MinIO operations are wrapped in try-catch blocks to handle connectivity issues, unfound buckets, or permission problems.

Contributing

Contributions are welcome! Please feel free to open issues or submit pull requests on the GitHub repository.

License

This package is licensed under the MIT License. See the LICENSE file for more details.