web3-storage
ipfs-car 🚘✨⬢
Convert files to content-addressable archives (.car) and back
Description
ipfs-car is a library and CLI tool to pack & unpack files from Content Addressable aRchives (CAR) file. A thin wrapper over @ipld/car and unix-fs.
Content-addressable archives store data as blocks (a sequence of bytes) each prefixed with the Content ID (CID) derived from the hash of the data; typically in a file with a .car extension.
Use ipfs-car to pack your files into a .car; a portable, verifiable, IPFS compatible archive.
$ ipfs-car pack path/to/files --output my-files.caror unpack files from a .car, and verify that every block matches it's CID
$ ipfs-car unpack my-files.car --output path/to/write/toFetch and locally verify files from a IPFS gateway over http
curl "https://ipfs.io/ipfs/bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu?format=car" | ipfs-car unpack -o imagesInstall
# install it as a dependency
$ npm i ipfs-car
# OR use the cli without installing via `npx`
$ npx ipfs-car --helpUsage
Pack files into a .car
# write a content addressed archive to stdout.
$ ipfs-car pack path/to/file/or/dir
# note: CAR data streamed to stdout will not have roots set in CAR header!
# specify the car file name.
$ ipfs-car pack path/to/files --output path/to/write/a.car
# by default, ipfs-car will wrap files in an IPFS directory.
# use --no-wrap to avoid this.
$ ipfs-car pack path/to/file --no-wrap --output path/to/write/a.carUnpack files from a .car
# unpack files to a specific path.
$ ipfs-car unpack path/to/my.car --output /path/to/unpack/files/to
# unpack a specific root.
$ ipfs-car unpack path/to/my.car --root <cid1>
# unpack files from a .car on stdin.
$ cat path/to/my.car | ipfs-car unpackShow the files and directories in a .car
# show the files and directories.
$ ipfs-car ls path/to/my.car
# show the files and directories, their CIDs and byte sizes.
$ ipfs-car ls path/to/my.car --verboseShow the root CIDs in a .car
# show the CID roots found in the CAR header.
$ ipfs-car roots path/to/my.car
# show the CID roots found implicitly from the blocks in the file.
$ ipfs-car roots --implicit path/to/my.carShow the block CIDs in a .car
# show the CIDs for all the blocks.
$ ipfs-car blocks path/to/my.carGet other information about a CAR
# generate CID for a CAR.
$ ipfs-car hash path/to/my.carAPI
To pack files into content-addressable archives, you can use the following:
createFileEncoderStreama factory function for creating aReadableStreamthat encodes a single file into DAGBlocks.createDirectoryEncoderStreama factory function for creating aReadableStreamfor encoding a directory of files into DAGBlocks.CAREncoderStreamaTransformStreamsub-class that you can writeBlocks to and readUint8ArrayCAR file data from.
To unpack content-addressable archives to files, you should use @ipld/car and ipfs-unixfs-exporter modules.
Examples
Basic single file pack
import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
const file = new Blob(['Hello ipfs-car!'])
const carStream = createFileEncoderStream(file).pipeThrough(new CAREncoderStream())
// carStream.pipeTo(somewhereWritable)Directory pack to file system in Node.js
import { Writable } from 'stream'
import { createDirectoryEncoderStream, CAREncoderStream } from 'ipfs-car'
import { filesFromPaths } from 'files-from-path'
const files = await filesFromPaths(process.argv.slice(2))
await createDirectoryEncoderStream(files)
.pipeThrough(new CAREncoderStream())
.pipeTo(Writable.toWeb(process.stdout))Usage: node script.js file0 file1 dir0 > my.car.
Obtaining the root CID
The root CID is the final block generated by the file/directory encoder stream. Use a transform stream to record the CID of the last block generated:
import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
const file = new Blob(['Hello ipfs-car!'])
let rootCID
await createFileEncoderStream(file)
.pipeThrough(new TransformStream({
transform (block, controller) {
rootCID = block.cid
controller.enqueue(block)
}
}))
.pipeThrough(new CAREncoderStream())
.pipeTo(new WritableStream())
console.log(rootCID.toString())Adding root CIDs to the CAR header
If you need root CIDs in the CAR header, there are two approaches you can use:
- Buffer all the DAG blocks, then encode with known root:
import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
const file = new Blob(['Hello ipfs-car!'])
const blocks = []
// buffer the output
await createFileEncoderStream(file)
.pipeTo(new WritableStream({ write: b => blocks.push(b) }))
const rootCID = blocks.at(-1).cid
const blockStream = new ReadableStream({
pull (controller) {
if (blocks.length) {
controller.enqueue(blocks.shift())
} else {
controller.close()
}
}
})
await blockStream
.pipeThrough(new CAREncoderStream([rootCID])) // pass root to CAR encoder
.pipeTo(new WritableStream())- Write to disk with placeholder CID, then update after DAG is completely generated (Note: Node.js only):
import fs from 'fs'
import { Writable } from 'stream'
import { CarWriter } from '@ipld/car/writer'
import { CID } from 'multiformats/cid'
import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
// Root CID written in CAR file header before it is updated with the real root CID.
const placeholderCID = CID.parse('bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi')
const file = new Blob(['Hello ipfs-car!'])
let rootCID
await createFileEncoderStream(file)
.pipeThrough(new TransformStream({
transform (block, controller) {
rootCID = block.cid
controller.enqueue(block)
}
}))
.pipeThrough(new CAREncoderStream(placeholderCID))
.pipeTo(Writable.toWeb(fs.createWriteStream('path/to/my.car')))
// update roots in CAR header
const fd = await fs.promises.open(opts.output, 'r+')
await CarWriter.updateRootsInFile(fd, [rootCID])
await fd.close()Unpacking files from a CAR
This functionality is not provided by this library, but is easy to do with @ipld/car and ipfs-unixfs-exporter modules:
import { CarIndexedReader } from '@ipld/car/indexed-reader'
import { recursive as exporter } from 'ipfs-unixfs-exporter'
const reader = await CarIndexedReader.fromFile('path/to/my.car')
const roots = await reader.getRoots()
const entries = exporter(roots[0], {
async get (cid) {
const block = await reader.get(cid)
return block.bytes
}
})
for await (const entry of entries) {
if (entry.type === 'file' || entry.type === 'raw') {
console.log('file', entry.path, entry.content)
} else if (entry.type === 'directory') {
console.log('directory', entry.path)
}
}Contributing
Feel free to join in. All welcome. Open an issue!
License
Dual-licensed under MIT + Apache 2.0