News Feed

• Alex fixed bugs in the postmodern backend and added new documentation.
• Ian promoted a new CLP backend which is an all-lisp solution based on cl-prevalence and cl-containers that is currently at Alpha quality. It requires the latest patches to cl-containers and a Mercurial fork of cl-prevalence which can be retrieved using 'hg clone http://bitbucket.org/skypher/cl-prevalence/'
• We are deprecating the CLSQL backend by moving it into contrib; it will not be a part of the 1.0 release. You can still use it by creating a symlink in the elephant root directory to the appropriate ele-xx files in src/contrib/rread/db-clsql

Download it now!

Bug fixes and minor feature enhancements

Significant development work has been taking place on the darcs elephant-unstable repository over the past year. The last stable release (0.9.1) is quite stale, lacking many bug fixes and features that are part of the upcoming 1.0 release.

We've decided to release a stable snapshot of the development tree and recommend that all users engaged in active development switch to this version of Elephant. This includes changing your target darcs repository from elephant-unstable to elephant-1.0.

While the Alpha 1 snapshot has not undergone rigorous testing, it is in daily use in a number of live applications and our test suites are currently green for all 0.9.1 and many 1.0 features on all data stores on SBCL/Linux. We invite users of other platforms to help expand the scope of testing of the 1.0 tree to ensure that any platform specific bugs are resolved prior to a final release.

The following new features have been introduced since our last release, 0.9.1, in November 2007.

New slot types

Hierarchical indexed slots

Indexed slots can now index a base class slot as well as all subclasses inheriting that slot. Index inheritance can be inhibited by redefining the slot in a base class.

Virtual derived index slots

A better way to manage derived indexes (the old derived index model is deprecated). You can define a virtual slot that is used to create the index. A slot argument defines the function of slot values that populates the index. Hints are available that inhibit updates to only occur when certain slots are written.

Set-valued slots

A simple API to an unordered aggregate slot using the pset api for access

Associations

Many-to-1 and many-to-many associations between slots in two different classes. They also uses a variation of the pset api.

'Checkout' slot caching

A simple means for runtime switching between persistent and transient slot behavior with the ability to sync transient values back to the store. Can buy up to 100x in read/write throughput at the cost of automatic thread safety

Dynamic schema evolution

• Global persistent instance map (all instances are reachable unless deleted)
• Each class has a DB schema that stores the class structure
• Multiple schema versions of a class may exist in the database concurrently
• Upgrade of DB instances to the latest schema can be done greed or lazily
• Dereferencing pointers to deleted instances returns nil; this is a lazy form of referential integrity.

Berkeley DB Store

• BDB 4.7 support (now recommended)
• Support to globally enable MVCC from my-config.sexp; :mvcc t to open-store and :snapshot t in with-transaction
• Read/write performance has been improved by up to 2x
• db_deadlock support deprecated in favor of built-in operators (no external process needed)

Postmodern Store

• Data caching within large transactions
• Caching between transactions on a per-thread basis

Miscellaneous

• Classes instances can exist in more than one store
• Multi-store operations are safer (transaction interleaving, for example)
• Serializer performance enhancements and bug fixes
• Migration bug fixes as well and upgrade support from 0.9.1
• Too many bug fixes and small enhancements to enumerate...

Garbage Collection (prototype)

The code base contains a working prototype of an online garbage collector using a mark and sweep strategy. Testing and documentation are not complete but you can experiment with it. Details are available on the mailing list archive from Jan 5th, 2009. Complete functionality will be available in the final release.

Query Language and Interpreter (prototype)

We also have a working sketch of a simple query language for elephant (query.lisp) that uses indices and a preliminary sketch of a query planner and interpreter (query2.lisp) that does something akin to joins. A modestly optimized form of this will be available as part of the final 1.0 release. Comments on the simple constraint language will be appreciated.

There is still much to do to iron down the final release:

• Associations still contain a few bugs, so we don't recommend depending on them yet.
• Derived index slots haven't been extensively used or tested
• Cached slots have not been extensively tested for corner cases, but do have some modest regression tests to validate their basic functionality and should be in good shape for single-threaded use
• Schema evolution There are also some minor problems that can crop up occasionally with schema evolution, but there are almost always workarounds so if you run into trouble e-mail elephant-devel
• Garbage collection has not been fully validated and should not be considered reliable
• Support a full constraint language specification in the query interface
• We need better test coverage (particularly new features)
• Documentation of new features

Ian Eslick has primary responsibility for the features in this release. Alex Mizrahi, Henrik Hjelte and Robert Read have put in significant work upgrading and improving the SQL backends. Special thanks to Leslie Polzer for various contributions and picking up some of the support slack.

Other noteable contributors to this release in the form of small patches, tracking down hairy bugs, suggestions and documentation include: Christoph Ludwig, Anton Kazennikov, Glen Tarcea, Frank Schorr, Daniel Salama, Lukas Giessmann, Patrick May, Quan Hu, Red Daly, Richard Szopa, Sean Ross, Tiarnan O'Corrain, and Elliott Slaughter. Our apologies if we have left anyone out (email eslick at media mit edu if we have).

Elephant 0.9.1 is now officially released.

This release provides official support for using the postmodern interface to Postgres as a back-end store. This back-end is about three times faster than the generic SQL interface on top of Postgres.

Note that existing Postgres repositories are not compatible with a Postmodern repository. However, Elephant supports multi-repository migration. If you have an existing Postgres repository and want to migrate to Postmodern, you should create a completely new Postmodern repository and migrate all of your data into it. The migration can be accomplished with a single command.

If you have been using the (unofficially released) postmodern backend, you will have to migrate your data into a CL-SQL based postgres backend, or into a BDB backend, and then migrate into your new postmodern repository. This release is not compatible with previous (development) versions of the postmodern functionality.

This release has been tested on a 64-bit Linux with SBCL 1.0.10, and other platforms as well. If anybody downloads it and tests it, please run the (large) set of automated tests and report your results to elphant-devel at common-lisp.net. We will update the platforms page as we receive your updates.

Download it now!

Elephant 0.9 is a significant improvement in stability, platform support and feature set over Elephant 0.6.0, our last major release. This release should be considered a stable release, and any major bugs that are uncovered will be accommodated by a patch release.

Major Features of Elephant 0.9

• Expanded and improved platform support
  • SBCL 1.0, Allegro 8.0/7.0, Lispworks 5.0 and OpenMCL 1.0
  • Elephant has been verified on every major OS (Linux, Mac OS X or Windows) for which these lisps are available
  • New support for 64-bit lisps and improved support for Unicode
  • Elephant is now thread-safe under both BDB and SQL data stores
  • A wide set of configurations have now been validated.
• New datastructures and interfaces
  • Persistent sets - lightweight collections with a simple API
  • Mapping operators - Efficient, cons free access to btrees, sets and class indices
• All new users manual
  • A new, basic tutorial incorporating the major new features
  • Expanded user guide for specific features
  • A new rough guide to the internal APIs for developers
• Other benefits or features
  • Simplified user configuration using a single file, my-config.sexp
  • Proper upgrades from 0.6, via the ability to open 0.6 databases in 0.9
  • The Berkeley DB backend is up-to-date with the latest library version and requires Berkeley DB 4.5
  • A host of performance improvements, bug fixes and minor feature enhancements (see the development list or prior announcements for some of the details)

Roadmap

We are planning a string of interim releases between 0.9 and 1.0, but we recommend that people using Elephant for active web sites and important applications use 0.9 until 1.0 is available. We will not guarantee clean upgrades between 0.9.x releases if the database format changes and cannot guarantee the same level of stability outside these major release milestones. The features planned for the 1.0 release of Elephant are recorded as tickets at the Trac project site.

Visit the documentation page to read the new reference manual and the download page to get the release.

With the marked improvement in stability and functionality of the code base, the maintainers have decided to abandon the 0.6.1 release label and change the upcoming release to 0.9. This change puts us on a new roadmap leading to a 1.0 release later in 2007.

New features and improvements in release 0.9 RC1

Additional features integrated since the Beta 0.6.1 announcement:

• Full platform support
  • SBCL 1.0, Allegro 8.0/7.0, Lispworks 5.0 and OpenMCL 1.0
  • All these lisps should work on every major OS (Linux, Mac OS X, Windows) they are released on
  • A wide set of configurations has been tested.
  • 64-bit support has been validated, but not recently and not on all platforms (testers welcome!)
• Mapping operators
  • BTrees, BTree indicies, class index and slot/derived indices all have a map operation
  • Enable mapping a function over values and ranges
  • Map in both ascending and descending order
  • Collect function results into a list
  • Efficient stack allocation where appropriate
• Persistent sets
  • A lighter-weight API for persistent collections
  • Provides insert, remove, find, map and list operations
  • With appropriate methods, makes one to many slots easy to construct, see tutorial and user-guide for details.
• Complete update and rewrite of the users manual
  • All new and up-to-date tutorial
  • New section containing more detailed implementation information for advanced users
  • New sections for prospective data store developers
  • Detailed installation instructions (supercedes INSTALL file)
  • A list of use case scenarios that should help developers understand Elephant better
  • Much work remains to clean up the manual for 1.0, but the bulk of the rewrite is done and we invite comment on what is missing or unclear.
• Feature tweaks
  • Drop commands for persistent collections to reclaim storage for reuse.
  • Expanded support for nested aggregates in serializer and migrate (i.e. an array of persistent objects will be properly migrated, before only btree collections of persistent objects were properly copied).
  • Better error handling during object creation, change class operations, etc.
  • Better error checking during cross-store operations.
  • Persistent instances passed to print-object print their oid by default to create a recognizable signature
  • And much more...
• Performance enhancements
  • Option to increase cache size for BDB backend
  • Improve contention performance while mapping btrees
  • Improved map performance
  • And much more...

Supported Platforms

• See the platform page for details
• OpenMCL will be verified as part of the beta cycle
• CMU should work, but there are no users so it is not officially supported

Download and Install

The beta release will only be available via CVS. Download the latest development HEAD.

cvs -z3 -d :pserver:anonymous:anonymous@common-lisp.net:/project/elephant/cvsroot checkout elephant

This will put the elephant release into the directory 'elephant'. Read the INSTALL, UPGRADE and UPGRADE-BDB files for further instructions.

Purpose of the beta release

The tree is effectively a release candidate. We only need to verify OpenMCL, beef up the documentation and examples and make sure we have good test coverage. Otherwise this code base appears stable enough to release.

New features in 0.6.1

The beta release is a more stable version of the feature set in the alpha release. A list of the changes between alpha and beta can be found in the doc/oldfiles/TODO file.

Supported platforms:

• SBCL, Allegro, CMU, OpenMCL, Lispworks (build system is not automated for Windows)
• CMU, OpenMCL and Lispworks have not been fully tested and may require minor bug fixes

Download and Install

The alpha release will only be available via CVS

cvs -z3 -d :pserver:anonymous:anonymous@common-lisp.net:/project/elephant/cvsroot checkout -r ELEPHANT-0-6-1-alpha elephant

This will put the elephant release into the directory: elephant. Read the INSTALL, UPGRADE and UPGRADE-BDB files for further instructions.

Purpose of the alpha release

The primary developers do not have ready access to all supported platforms and would like to ask the community to help validate the current implementation on other platforms. We also are using the alpha to improve up our test suite. For more details please see the TODO file. If you have a favorite feature such as multi-threading, 64-bit, etc., please free to submit a test for the test suite.

New features in 0.6.1

• Simplified build and site configuration support:
  • See config.sexp in root directory for site customization (no more editing code files)
  • Linux and Mac systems should automatically build libraries when asdf is called
• 64-bit lisps are now supported
• 64-bit and 32-bit lisps can read and run off the same database files (on the same machine)
• Multithreading
• Elephant should now be thread safe, including sharing a single store-controller across threads
• Read comments in src/elephant/transactions.lisp and BDB users should read src/db-bdb/bdb-transactions.lisp
• Improved support for mixing transactions and store-controllers
• Upgrading
• Elephant 0.6.1 can open and directly manipulate 0.6.0 databases
• Existing 0.6.0 databases can be upgraded via the 'upgrade' function which takes
• Upgrading is required for 64-bit systems
• New serializer-independant metadata should enable future upgrades easier
• NOTE: There may be some problems upgrading SQL databases
• Berkeley DB backend
• BDB 4.5 required (see UPGRADE-BDB)
• *auto-commit* is no longer required. All data access methods auto-commit if there is no active transaction
• store-controller accepts :deadlock-detect keyword (t or nil) which will run db_deadlock as a background process to abort deadlocked threads
• optimize-storage is a new store-controller method currently supported by BDB backend. It compacts a whole database or only a specific BTree and returns free pages to the free list or to the file system.
• Minor features
• Re-organization to the internal structure in this release
• Renaming including removing defunct sleepycat naming scheme, backend packages, etc.
• The serializer was modularized to allow future releases to change serializer
• strategies or implement custom serializers and to be able to open legacy databases
• Remove various warnings in SBCL build, etc.
• Separated utilities into their own package and directory
• Serializer improvements
• 0.6.1 databases can be shared across lisps running on hardware of the same endianness
  (i.e. all x86 platforms or PPC/Alpha, etc)
• Simplified unicode serialization support across all platforms
• Performance improvements in multi-threading situations
• Pushing :elephant-without-optimize onto *features* will disable optimization declarations and simplifying debugging
• Other fixes and features documented in the TODO file

See the downloads page for more information.

The latest version is 0.6.0, is offically released as of May 27, 2006. It has been tested on a variety of lisp implementations and with a variety of storage repositories. This release is mainly the work of Ian Eslick, and offers a significant refactoring as well as the very convenient feature of automatic slot indexing. Julian Stecklina submitted a patch for FreeBSD which is in the CVS repository now but not in the 0.6.0 release.