Amazon S3¶

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

from S3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN

# see http://developer.yahoo.com/performance/rules.html#expires
AWS_HEADERS = {
    'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT',
    'Cache-Control': 'max-age=86400',
}

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

>>> from django.core.files.storage import default_storage
>>> print default_storage.__class__
<class 'S3Storage.S3Storage'>

>>> from django.core.files.storage import default_storage
>>> print default_storage.connection
S3Connection:s3.amazonaws.com

>>> from django.db import models
>>> class Resume(models.Model):
...     pdf = models.FileField(upload_to='pdfs')
...     photos = models.ImageField(upload_to='photos')
...
>>> resume = Resume()
>>> print resume.pdf.storage
<S3Storage.S3Storage object at ...>

>>> from django.core.files.storage import default_storage
>>> from django.core.files.base import ContentFile
>>> from django.core.cache import cache
>>> from models import MyStorage

>>> default_storage.exists('storage_test')
False
>>> file = default_storage.open('storage_test', 'w')
>>> file.write('storage contents')
>>> file.close()

>>> default_storage.exists('storage_test')
True
>>> file = default_storage.open('storage_test', 'r')
>>> file.read()
'storage contents'
>>> file.close()

>>> default_storage.delete('storage_test')
>>> default_storage.exists('storage_test')
False

>>> obj1 = MyStorage()
>>> obj1.normal
<FieldFile: None>
>>> obj1.normal.size
Traceback (most recent call last):
...
ValueError: The 'normal' attribute has no file associated with it.

>>> obj1.normal.save('django_test.txt', ContentFile('content'))
>>> obj1.normal
<FieldFile: tests/django_test.txt>
>>> obj1.normal.size
7
>>> obj1.normal.read()
'content'

>>> obj1.normal.open()
>>> obj1.normal.read(3)
'con'
>>> obj1.normal.read()
'tent'
>>> '-'.join(obj1.normal.chunks(chunk_size=2))
'co-nt-en-t'

>>> obj2 = MyStorage()
>>> obj2.normal.save('django_test.txt', ContentFile('more content'))
>>> obj2.normal
<FieldFile: tests/django_test_.txt>
>>> obj2.normal.size
12

>>> cache.set('obj1', obj1)
>>> cache.set('obj2', obj2)
>>> cache.get('obj2').normal
<FieldFile: tests/django_test_.txt>

>>> obj2.delete()
>>> obj2.normal.save('django_test.txt', ContentFile('more content'))
>>> obj2.normal
<FieldFile: tests/django_test_.txt>

>>> obj3 = MyStorage.objects.create()
>>> obj3.default
<FieldFile: tests/default.txt>
>>> obj3.default.read()
'default content'

>>> obj3.delete()
>>> obj3 = MyStorage()
>>> obj3.default.read()
'default content'

>>> obj4 = MyStorage()
>>> obj4.random.save('random_file', ContentFile('random content'))
>>> obj4.random
<FieldFile: .../random_file>

>>> obj1.normal.delete()
>>> obj2.normal.delete()
>>> obj3.default.delete()
>>> obj4.random.delete()

Apache Libcloud¶

Apache Libcloud is an API wrapper around a range of cloud storage providers. It aims to provide a consistent API for dealing with cloud storage (and, more broadly, the many other services provided by cloud providers, such as device provisioning, load balancer configuration, and DNS configuration).

As of v0.10.1, Libcloud supports the following cloud storage providers:

• Amazon S3
• Google Cloud Storage
• Nimbus.io
• Ninefold Cloud Storage
• Rackspace CloudFiles

Libcloud can also be configured with relatively little effort to support any provider using EMC Atmos storage, or the OpenStack API.

LIBCLOUD_PROVIDERS = {
    'amazon_1': {
        'type': 'libcloud.storage.types.Provider.S3_US_STANDARD_HOST',
        'user': '<your username here>',
        'key': '<your key here>',
        'bucket': 'bucket-1',
    },
    'amazon_2': {
        'type': 'libcloud.storage.types.Provider.S3_US_STANDARD_HOST',
        'user': '<your username here>',
        'key': '<your key here>',
        'bucket': 'bucket-2',
    },
    'google': {
        'type': 'libcloud.storage.types.GOOGLE_STORAGE',
        'user': '<Your Google APIv1 username>',
        'key': '<Your Google APIv1 Key>',
        'bucket': 'bucket-3',
    },
}

>>> from storages.backends.apache_libcloud import LibCloudStorage
>>> store = LibCloudStorage('amazon_1')
>>> store.driver.create_container('bucket-1')

DEFAULT_LIBCLOUD_PROVIDER = 'amazon-1'

LIBCLOUD_PROVIDERS = {
    'default': {
        'type': ...
    },
}

DEFAULT_FILE_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage'

>>> from storages.backends.apache_libcloud import LibCloudStorage
>>> store = LibCloudStorage('amazon_1')
Traceback (most recent call last):
...
ImproperlyConfigured: Unable to create libcloud driver type libcloud.storage.types.Provider.S3_US_STANDARD_HOST: No CA Certificates were found in CA_CERTS_PATH.

import libcloud.security
libcloud.security.CA_CERTS_PATH.append("/path/to/your/cacerts.pem")

Azure Storage¶

A custom storage system for Django using Windows Azure Storage backend.

DEFAULT_FILE_STORAGE = 'storages.backends.azure_storage.AzureStorage'

AZURE_ACCOUNT_NAME = "azure_account_name"

CouchDB¶

A custom storage system for Django with CouchDB backend.

Database¶

Class DatabaseStorage can be used with either FileField or ImageField. It can be used to map filenames to database blobs: so you have to use it with a special additional table created manually. The table should contain a pk-column for filenames (better to use the same type that FileField uses: nvarchar(100)), blob field (image type for example) and size field (bigint). You can’t just create blob column in the same table, where you defined FileField, since there is no way to find required row in the save() method. Also size field is required to obtain better perfomance (see size() method).

So you can use it with different FileFields and even with different “upload_to” variables used. Thus it implements a kind of root filesystem, where you can define dirs using “upload_to” with FileField and store any files in these dirs.

It uses either settings.DB_FILES_URL or constructor param ‘base_url’ (see __init__()) to create urls to files. Base url should be mapped to view that provides access to files. To store files in the same table, where FileField is defined you have to define your own field and provide extra argument (e.g. pk) to save().

Raw sql is used for all operations. In constructor or in DB_FILES of settings.py () you should specify a dictionary with db_table, fname_column, blob_column, size_column and ‘base_url’. For example I just put to the settings.py the following line:

DB_FILES = {
    'db_table': 'FILES',
    'fname_column':  'FILE_NAME',
    'blob_column': 'BLOB',
    'size_column': 'SIZE',
    'base_url': 'http://localhost/dbfiles/'
}

And use it with ImageField as following:

player_photo = models.ImageField(upload_to="player_photos", storage=DatabaseStorage() )

DatabaseStorage class uses your settings.py file to perform custom connection to your database.

The reason to use custom connection: http://code.djangoproject.com/ticket/5135 Connection string looks like:

cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;DATABASE=testdb;UID=me;PWD=pass')

It’s based on pyodbc module, so can be used with any database supported by pyodbc. I’ve tested it with MS Sql Express 2005.

Note: It returns special path, which should be mapped to special view, which returns requested file:

def image_view(request, filename):
    import os
    from django.http import HttpResponse
    from django.conf import settings
    from django.utils._os import safe_join
    from filestorage import DatabaseStorage
    from django.core.exceptions import ObjectDoesNotExist

    storage = DatabaseStorage()

    try:
        image_file = storage.open(filename, 'rb')
        file_content = image_file.read()
    except:
        filename = 'no_image.gif'
        path = safe_join(os.path.abspath(settings.MEDIA_ROOT), filename)
        if not os.path.exists(path):
            raise ObjectDoesNotExist
        no_image = open(path, 'rb')
        file_content = no_image.read()

    response = HttpResponse(file_content, mimetype="image/jpeg")
    response['Content-Disposition'] = 'inline; filename=%s'%filename
    return response

Note: If filename exist, blob will be overwritten, to change this remove get_available_name(self, name), so Storage.get_available_name(self, name) will be used to generate new filename.

FTP¶

This implementation was done preliminary for upload files in admin to remote FTP location and read them back on site by HTTP. It was tested mostly in this configuration, so read/write using FTPStorageFile class may break.

Image¶

A custom FileSystemStorage made for normalizing extensions. It lets PIL look at the file to determine the format and append an always lower-case extension based on the results.

MogileFS¶

This storage allows you to use MogileFS, it comes from this blog post.

The MogileFS storage backend is fairly simple: it uses URLs (or, rather, parts of URLs) as keys into the mogile database. When the user requests a file stored by mogile (say, an avatar), the URL gets passed to a view which, using a client to the mogile tracker, retrieves the “correct” path (the path that points to the actual file data). The view will then either return the path(s) to perlbal to reproxy, or, if you’re not using perlbal to reproxy (which you should), it serves the data of the file directly from django.

• MOGILEFS_DOMAIN: The mogile domain that files should read from/written to, e.g “production”
• MOGILEFS_TRACKERS: A list of trackers to connect to, e.g. [“foo.sample.com:7001”, “bar.sample.com:7001”]
• MOGILEFS_MEDIA_URL (optional): The prefix for URLs that point to mogile files. This is used in a similar way to MEDIA_URL, e.g. “/mogilefs/”
• SERVE_WITH_PERLBAL: Boolean that, when True, will pass the paths back in the response in the X-REPROXY-URL header. If False, django will serve all mogile media files itself (bad idea for production, but useful if you’re testing on a setup that doesn’t have perlbal running)
• DEFAULT_FILE_STORAGE: This is the class that’s used for the backend. You’ll want to set this to project.app.storages.MogileFSStorage (or wherever you’ve installed the backend)

from django.core.filestorage import storage

# TODO: Find a better way to deal with classes. Maybe a generator?
class AvatarStorage(storage.__class__):
    mogile_class = 'avatar'

class ThumbnailStorage(storage.__class__):
    mogile_class = 'thumb'

class Avatar(models.Model):
    user = models.ForeignKey(User, null=True, blank=True)
    image = models.ImageField(storage=AvatarStorage())
    thumb = models.ImageField(storage=ThumbnailStorage())

urlpatterns += patterns(",
    (r'^%s(?P<key>.*)' % settings.MOGILEFS_MEDIA_URL[1:],
        'MogileFSStorage.serve_mogilefs_file')
)

MongoDB¶

A GridFS backend that works with django_mongodb_engine and the upcoming GSoC 2010 MongoDB backend which gets developed by Alex Gaynor.

Usage (in settings.py):

DATABASES = {
    'default': {
        'ENGINE': 'django_mongodb_engine.mongodb',
        'NAME': 'test',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': 27017,
        'SUPPORTS_TRANSACTIONS': False,
    }
}

DEFAULT_FILE_STORAGE = 'storages.backends.mongodb.GridFSStorage'
GRIDFS_DATABASE = 'default'

Overwrite¶

This is a simple implementation overwrite of the FileSystemStorage. It removes the addition of an ‘_’ to the filename if the file already exists in the storage system. I needed a model in the admin area to act exactly like a file system (overwriting the file if it already exists).

SFTP¶

Take a look at the top of the backend’s file for the documentation.

Symlink or copy¶

Stores symlinks to files instead of actual files whenever possible

When a file that’s being saved is currently stored in the symlink_within directory, then symlink the file. Otherwise, copy the file.