S3 destination support
Overview
Writing to Amazon S3-compatible file storage services such as AWS and Filebase is supported via the S3Dest
class from the @subsquid/file-store-s3
package. Use it by setting the dest
field of the Database
constructor argument to its instance. Constructor of S3Dest
accepts the following arguments:
url: string
: S3 URL in thes3://bucket/path
format.optionsOrClient?: S3Client | S3ClientConfig
: an optional S3 client or client config. By default, a simple config parameterized by environment variables is used:{
region: process.env.S3_REGION,
endpoint: process.env.S3_ENDPOINT,
credentials: {
accessKeyId: assertNotNull(process.env.S3_ACCESS_KEY_ID),
secretAccessKey: assertNotNull(process.env.S3_SECRET_ACCESS_KEY),
},
}
Example
This saves the processor data in the transfers-data
folder of the subsquid-testing-bucket
bucket at the Filebase service. The service only has one region and one endpoint, and here they are hardcoded to reduce the number of required envirionment variables and illustrate how connection parameters can be supplied programmatically. Full squid code is available in this repo.
import {Database} from '@subsquid/file-store'
import {S3Dest} from '@subsquid/file-store-s3'
import {assertNotNull} from '@subsquid/util-internal' // pulled by @subsquid/file-store-s3
...
const dbOptions = {
...
dest: new S3Dest(
's3://subsquid-testing-bucket/transfers-data',
{
region: 'us-east-1',
endpoint: 'https://s3.filebase.com',
credentials: {
accessKeyId: assertNotNull(process.env.S3_ACCESS_KEY_ID),
secretAccessKey: assertNotNull(process.env.S3_SECRET_ACCESS_KEY)
}
}
),
...
}
processor.run(new Database(dbOptions), async (ctx) => {
...
}