S3 Backend

class keg_storage.backends.s3.S3Storage(bucket, aws_region, aws_access_key_id=None, aws_secret_access_key=None, aws_profile=None, name='s3')[source]
delete(path)[source]

Delete the remote file specified by path.

list(path)[source]

Returns a list of ListEntry`s representing files available under the directory or prefix given in `path.

open(path: str, mode: Union[keg_storage.backends.base.FileMode, str])[source]

Returns a instance of RemoteFile for the given path that can be used for reading and/or writing depending on the mode given.

class keg_storage.backends.s3.S3Reader(bucket, filename, client)[source]
close()[source]

Cleanup and deallocate any held resources. This method may be called multiple times on a single instance. If the file was already closed, this method should do nothing.

read(size: int)[source]

Read and return up to size bytes from the remote file. If the end of the file is reached this should return an empty bytes string.

class keg_storage.backends.s3.S3Writer(bucket, filename, client, chunk_size=10485760)[source]

Writes to S3 are quite a bit more complicated than reads. To support large files, we cannot write in a single operation and the API does not encourage streaming writes so we make use of the multipart API methods.

The process can be summarized as:
  • Create a multipart upload and get an upload key to use with subsequent calls.
  • Upload “parts” of the file using the upload key and get back an ID for each part.
  • Combine the parts using the upload key and all the part IDs from the above steps.

The chunked nature of the uploads should be mostly invisible to the caller since S3Writer maintains a local buffer.

Because creating a multipart upload itself has an actual cost and there is no guarantee that anything will actually be written, we initialize the multipart upload lazily.

abort()[source]

Use if for some reason you want to discard all the data written and not create an S3 object

close()[source]

Cleanup and deallocate any held resources. This method may be called multiple times on a single instance. If the file was already closed, this method should do nothing.

write(data: bytes)[source]

Write the data buffer to the remote file.

class keg_storage.backends.s3.S3FileBase(mode, bucket, filename, client)[source]

Read and write operations for S3 are very different so individual subclasses are used for each. Read+Write mode is not available for this backend.