Berkeley DB Java Edition 3.3.75 Change Log
Please note that JE now requires Java 1.5 or higher. Java 1.4 is no
longer supported.
Log File On-Disk Format Changes:
JE 3.3 has moved to on-disk file format 6.
The change is forward compatible in that JE files created with release
3.2 and earlier can be read when opened with JE
3.3. The change is not backward compatible in that files
created with JE 3.3 cannot be read by earlier releases.
Note that if an existing environment is opened read/write, a new log
file is written by JE 3.3 and the environment can no
longer be read by earlier releases.
A number of these on-disk format changes were made to reduce
disk-space usage. [#15399]
Also note the binary and compile time incompatibilities between JE 3.3
and JE 3.2 described in the API Changes sections.
The changes between 3.2 and 3.3.75 are described below. The
changes are listed separately for each patch release, with the most recent
patch release at the top.
Changes in 3.3.75
General Environment Changes
- Fix a bug that caused the space taken by internal metadata in JE log files
to increase over a long period of time. The rate of increase was slow in most
cases, but in at least one observed case became rapid after a long time period
and after the log cleaner became backlogged. To determine whether your JE log
exhibits this problem, run
java -jar je.x.y.z.jar DbPrintLog -h DIR -S
and examine the line labeled MapLN on the left. If the amount of the log taken
by MapLNs is 10% or greater, or if you see this number increasing steadily over
time, then your application is probably experiencing this problem.
By installing JE 3.3.75 or later, the excess disk space will automatically
be reclaimed over time, as ordinary checkpoints and log cleaning occur. If you
wish to recreate your database rather than wait for this to occur gradually,
you can use DbDump and DbLoad to do so.
We'd like to express our appreciation and sincere thanks to Jules and the
other folks at Xoopit who reported
this problem and helped us to diagnose it. We would not have found or fixed
this problem as quickly as we did without their help.
[#16610]
- The Environment.scanLog method and related classes (LogScanner and
LogScanConfig), have been removed from the public API. This API was added for
s single customer with a special purpose requirement that is no longer
applicable. This API is now private and not present in the documentation.
[#16453]
Changes in 3.3.74
General Environment Changes
- Fix a problem that could cause LogFileNotFoundException in large databases.
This could cause data loss and therefore all users are strongly recommended to
upgrade to JE 3.3.74. This problem is found in all prior releases of JE.
This bug can occur only for databases with a Btree that is 4 or more levels
deep. For the default maximum entries per Btree node (128), this means the
database must have grown to at least 10 million records to be a candidate for
occurrance of this problem. This is only an approximation and may be larger or
smaller if DatabaseConfig.setNodeMaxEntries has been called.
[#16523]
- Fix a problem where a static field instance in JE could continue to
maintain a reference to internal data structures after an
Environment
is fully closed, preventing the Java GC from
reclaiming the memory for the closed environment. This can occur when locking
is disabled. This problem was originally mentioned in this forum
post. [#16453]
-
Fix a bug that caused NullPointerException during recovery (Environment
construction) after deleting records in a database and then truncating or
removing the database, as in the following stack trace:
Caused by: java.lang.NullPointerException
at com.sleepycat.je.recovery.RecoveryManager.redoUtilizationInfo(RecoveryManager.java:2276)
at com.sleepycat.je.recovery.RecoveryManager.redoLNs(RecoveryManager.java:1247)
[#16515]
-
Fixed a bug in which RandomAccessFile.length() not being thread-safe can
potentially cause unexpected resutls. [#16440]
Direct Persistence Layer (DPL), Collections and Bind packages
-
Fix a bug that under certain circumstances causes "IllegalArgumentException:
Not a key class" when calling EntityStore.getSubclassIndex,
EntityStore.getPrimaryConfig, EntityStore.getSecondaryConfig, or
PrimaryIndex.put, and a composite key class is used. [#16407]
-
Fix a bug that causes class evolution to occur when an instance of an entity
subclass is stored for the first time, after storing an instance of the entity
(base) class. In some circumstances this could result in the following
exception:
Caused by: java.lang.NullPointerException
at com.sleepycat.persist.impl.ComplexFormat.newInstance(ComplexFormat.java:478)
at com.sleepycat.persist.impl.ComplexFormat$EvolveReader.newInstance(ComplexFormat.java:2003)
at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(PersistEntityBinding.java:88)
at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(PersistEntityBinding.java:60)
at com.sleepycat.persist.PrimaryIndex.get(PrimaryIndex.java:518
This exception could also be caused by two class changes that evolve the entity
twice, such that the second change reverts the first change to the original
version of the class. In other words, this could occur when there are three
versions of a class A, A-1, A-2 and A-3, and A-3 is identical to A-1.
Thanks to jhalex for reporting the problem in this
OTN forum
thread and helping to diagnose it. As mentioned in the forum thread, a
workaround for this problem is to call EntityModel.registerClass for all
entity subclasses.
Configuration, Documentation, Test and Build Changes:
-
Clarify the javadoc for Database.put() and Cursor.put(). Change the
phrase "If the key already appears in the database and duplicates are
not configured, the existing key/data pair will be replaced." to
"If the key already appears in the database and duplicates are not
configured, the data associated with the key will be replaced."
Changes in 3.3.69
General Environment Changes
- Fix a bug that sometimes causes Environment.truncateDatabase and
removeDatabase to hang, after opening an Environment with existing records in
that Database. [#16329]
- Fix a bug that leaves the je.lck file open when an Environment cannot be
opened for write access because another process has the same environment open
for write access. When this occurs, the new EnvironmentLockedException is now
thrown instead of an internal subclass of DatabaseException. [#16348]
- Fix a bug that causes an exception during Environment.close and prints the
Environment stats to the console after the following message: "Local Cache
Usage = ...". This bug could have caused a small memory leak for some
applications that used the shared cache feature and that open and close many
environments. [#16368]
- If an attempt to call Environment.cleanLog is made on a read-only
Environment, IllegalStateException is now thrown. Before, the call to cleanLog
was allowed, but log cleaning did not function properly. [#16368]
- Implement the -c flag of DbVerify. Before, this flag was in the usage
command but was not implemented. Now, it causes DbVerify to check log cleaner
metadata. [#16368]
- Fix a bug that causes OutOfMemoryError in DbVerify, Database.verify
or Database.getStats, when the size of the data that is verified or examined
exceeds the cache size. [#16368]
- Fix a bug that prevents opening an Environment as read-only under certain
circumstances, or causes queries in a read-only Environment to return out of
date, and possibly transactionally incorrect, data. The
LogFileNotFoundException may be thrown when the problem occurs. [#16368]
- Fix a problem that reduces concurrency for multi-threaded applications
using the Serializable isolation level. Before, when a reader performed an
exact search for a record by key and the record was found, writers were blocked
from inserting records past the last record in the database. Now, this
blocking does not occur. Thanks to OTN user Alfranio who found this problem
and identified the incorrect JE source code in this forum
post. [#16393]
Direct Persistence Layer (DPL), Collections and Bind packages
-
Fix a bug that prevents evolution of @SecondaryKey information in an
entity subclass (a class that extends an @Entity class). The fix applies to
adding, removing and renaming secondary key fields in an entity subclass.
The problem was reported in
this
forum thread. [#16253]
-
Add the ability to make fields persistent or not persistent independently of
the use of the
transient
keywork for Java serialization. To
override the normal rules for DPL field persistence, the @NotPersistent and
@NotTransient annotations may be specified. Normally these keywords are not
needed because the transient keyword is sufficient to mark a field as not
persistent. However, some applications may wish for a field to be transient
with respect to the DPL but not transient with respect to Java serialization,
or vice-versa. The new annotations are provided for such applications. The
feature was requested in
this
forum thread. [#16297]
-
Add
EntityCursor.setCacheMode
and getCacheMode
.
See the com.sleepycat.je.CacheMode
class for more information.
[#16239]
-
The
StoredMap
class now implements the standard
java.util.concurrent.ConcurrentMap
interface. In the 3.3.62
release, the ConcurrentMap
methods were implemented by the
StoredMap
class, but it was not declared to implement the
ConcurrentMap
interface. [#15382]
Changes in 3.3.62
New Features:
-
JE can now be configured to use a shared cache among multiple
environments. By using the shared cache, multiple open environments in
the same process will make better use of memory because the cache LRU
algorithm is applied across all information in all environments
sharing the cache. See EnvironmentConfig.setSharedCache() for more
information. Two related stats were also added to EnvironmentStats:
getNSharedCacheEnvironments and getSharedCacheTotalBytes. [#15267]
-
JE now provides a "temporary" database mode, specified through
DatabaseConfig.setTemporary() and StoreConfig.setTemporary(). Temporary
databases operate internally in deferred-write mode to provide reduced disk I/O
and increased concurrency. But unlike an ordinary deferred-write database, the
information in a temporary database is not durable or persistent, and therefore
temporary databases are lighter weight than deferred-write databases. More
details can be found in the Getting Started Guide and in the javadoc for
DatabaseConfig and StoreConfig.
In addition, two internal changes have been made to durable deferred-write
database behavior. These changes are compatible with the existing use of
deferred-write database, but may influence performance characteristics. These
changes were necessary to ensure that the log cleaner operates properly for
deferred-write databases.
- The JE checkpointer now effectively performs a Database.sync for
all open deferred-write databases. If you don't want this behavior
you can a) use a Temporary DB instead, or b) disable the
checkpointer. Disabling the checkpointer is commonly used to reduce
logging during a bulk load.
- JE now calls Database.sync for deferred-write databases when
they are closed. If the data is not intended to be durable, we
recommend using a Temporary DB instead.
[#15365]
-
Added a new EnvironmentConfig parameter: je.cleaner.upgradeToLogVersion. This
parameter is used to perform background upgrading of earlier format JE log
files. All log files having a log version prior to the specified version will
be cleaned -- migrating the log records forward to the end of the log and
updating their format -- at a time when no other log cleaning is necessary.
This parameter is intended for use in upgrading earlier format log files
forward to the current log format version, for example, to take advantage of
disk usage improvements. Note that earlier format log files are supported by
later JE releases and log upgrading is optional. [#15365]
-
JE now supports key prefixing. When enabled, the keys stored on a
given internal b-tree nodes may be represented as a set of
prefix/suffix combinations where the prefix is the common prefix of
all keys for the internal b-tree node. This can improve storage
efficiency for some key sets, both on-disk and in-memory, at the cost
of computational efficiency. The degree of storage improvement is a
function of the application's key values and access pattern. Key
prefixing can be enabled with the
DatabaseConfig.setKeyPrefixing(boolean)
method. [#15399]
-
Added the ability to control caching policy on a per-operation basis.
See Cursor.get/setCacheMode() and com.sleepycat.je.CacheMode to see
how calls to Cursor "get" and "put" operations can specify LRU, MRU,
or "pinned" cache hotness. [#15850]
-
New stats have been added to meter random vs sequential disk io.
EnvironmentStats.getNRandomReads()
EnvironmentStats.getNRandomWrites()
EnvironmentStats.getNRandomReadBytes()
EnvironmentStats.getNRandomWriteBytes()
EnvironmentStats.getNSequentialReads()
EnvironmentStats.getNSequentialWrites()
EnvironmentStats.getNSequentialReadBytes()
EnvironmentStats.getNSequentialWriteBytes()
return operation and byte counts for the number of random and
sequential disk IOs. All values are approximate and may differ from
the actual number of operations/byte-counts depending on the type of
disks and file system, disk geometry, and file system cache size.
[#16086]
-
A new stat has been added to gauge the cost of log file opens, which
can be a factor for applications with large data sets. A large value
for
EnvironmentStats.getNFileOpens()
can indicate a need
to tune the size of the file handle cache through increasing the
je.log.fileCacheSize
property. Also, general improvements
have been made to improve the file handle cache, and to remove a
bottleneck in concurrency in that area. [#16166]
General Environment Changes
-
JE btree nodes now use shared latches for a portion of the btree
access path. This used to be available through the
je.env.sharedLatches configuration property, and is now the
default. This results in better concurrency for read operations.
The je.env.sharedLatches parameter has been deprecated. [#12136]
-
In the past, JE's cache management pinned the internal metadata
representing application databases. That made the use of large numbers
of databases problematic, because significant portions of the cache
could be consumed by metadata. That has now been resolved, and
there are no cache or performance limitations on the number of
databases in a JE environment. [#13415]
-
The per-lock memory overhead has been reduced. [#15512]
-
Various parameters have been adjusted so that JE will run under the Google
Android platform. Refer to the Android HOWTO
document for details.
-
Fixed a bug in Transaction.setLockTimeout and setTxnTimeout so that the timeout
is applied for all isolation modes. Before, the timeout was not applied for
ReadCommitted and Serializable isolation under certain circumstances. The
problem was initially reported on this JE Forum
thread. [#16017]
-
Fixed a bug where an internal data structure was being accessed sequentially
during logging, while blocking other threads that attempt to write. For
applications where the data set does NOT fit in cache, the data structure was
small and the sequential access had little impact. But for applications where
the entire data set DOES fit in cache, the sequential access was causing
noticeable performance and concurrency problems during writing. [#16037]
-
Improved performance for applications having a very large JE cache and where
the active data set fits in the cache. Before, in this scenario, checkpoints
could take a long time to occur. This in turn prevented log cleaning from
being performed until after the end of the checkpoint. Two changes were made:
- Cleaner utilization information is now updated during the checkpoint. The
log cleaner will no longer wait until the end of the checkpoint to perform
cleaning. Note that although cleaning takes place during the checkpoint, log
files are still not deleted until the end of the checkpoint.
- A new environment configuration parameter -- je.checkpointer.highPriority
-- can be used to reduce the time to perform a checkpoint. If set to true, the
checkpointer uses more resources in order to complete the checkpoint in a
shorter time interval. Btree latches are held and other threads are blocked
for a longer period. Log cleaner record migration is performed by cleaner
threads instead of during checkpoints. A shorter checkpoint is beneficial
because a shorter recovery period will be required in the event of a crash.
However, application response time may be longer during the checkpoint, and
more cleaner threads may be required to maintain the configured
je.cleaner.minUtilization.
[#16037]
-
Improved concurrency when a large number of log files (more than fit in the
open-file-cache) are being accessed. Before, the latch on the cache was held
while opening the log file and reading the header. Now, only the latch on the
individual file is held. Note that the size of the open-file-cache is
configured with the je.log.fileCacheSize Environment configuration parameter.
Normally the cache can be configured to a size large enough to hold all files
being accessed. However, for very large data sets, this may not be practical.
[#16166]
-
The EnvironmentStats.getTotalLogSize statistic is now a slow stat (not returned
when StatsConfig.setFast(true) is called) and a bug was fixed that caused the
total to be incorrect and overflow in some cases. [#15985]
-
Fixed a bug where lock timeouts (DeadlockException) would sometimes occur after
twice the configured lock timeout interval. This fix may cause
DeadlockException to occur more frequently in some applications because the
configured lock timeout is now applied correctly. Note that the lock timeout
is 0.5 seconds by default and may be configured via the setLockTimeout method
in EnvironmentConfig and Transaction. [#16021]
Direct Persistence Layer (DPL), Collections and Bind packages
-
Several validation checks have been added or corrected having to do with entity
subclasses, which are @Persistent classes that extend an @Entity class.
- An IllegalArgumentException is now thrown when @PrimaryKey appears on an
entity subclass. Before this was documented to be illegal but was not
checked. [#15757]
- The DPL now throws IllegalArgumentException if an instance of an entity
subclass is embedded in another entity. It has always been illegal to embed
entities within other entities, but the check for entity subclasses was not
correct and has been fixed. [#16077]
- In certain circumstances a NullPointerException was thrown before when
attempting to embed an entity subclass, and this has also been corrected to
throw IllegalArgumentException. [#16077]
-
Optimize DPL marshaling for large numbers of embedded objects. Before, a
sequential lookup was used to manage the object graph. Now, a HashMap is used.
Thanks to user624180 who suggested this in a forum
post. [#16198]
-
Fix a DPL bug that caused exceptions when using a class Converter for an
instance containing non-simple fields. The problem was reported in
this
forum thread. Thanks to Stan Livitski for submitting a reproducible test
case. [#16233]
-
The StoredMap class now implements the Java 1.5 ConcurrentMap interface. Thanks
to jahlborn for proposing this change in
this
OTN forum thread. [#16218]
-
Add Java 1.5 generic type parameters where appropriate: EntryBinding,
EntityBinding and all binding implementations; all references to the Map and
Collection interfaces; etc.
API Changes:
-
Deprecated the EnvironmentStats.getCacheDataBytes method. The new getDataBytes
should be used instead. The getLockBytes method now includes the lock and
transaction overhead. Before it returned only the lock overhead. [#15267]
-
A number of fields in EnvironmentStats and TransactionsStats were
changed from int to long to prevent overflows, which resulted in a
change to the signature of a number of getXXX() methods. Regrettably,
this constitutes an incompatible binary and compile time change from
JE 3.2 to JE 3.3 for applications which used those getter
methods. There is no incompatibility if your application merely
instantiates an EnvironmentStat or TransactionStat or calls their
toString() methods. [#15979]
-
JE now requires Java 5 and uses generic types in method
signatures. Most notably for the user, the methods which define custom
comparators have been changed to these signatures:
DatabaseConfig.setBtreeComparator(Comparator<byte[]>)
DatabaseConfig.setBtreeComparator(Class<? extends
Comparator<byte[]>>)
DatabaseConfig.setDuplicateComparator<Comparator<byte[]>>
DatabaseConfig.setDuplicateComparator(Class<? extends
Comparator<byte[]>>)
If the application used a custom comparator defined as
public class MyCompare implements Comparator, Serializable
that should now be declared as
public class MyCompare implements Comparator<byte[]>, Serializable
Please note that while these are fine:
Comparator<byte[]> compareInstance = new MyCompare();
dbConfig.setBtreeComparator(compareInstance);
or
dbConfig.setBtreeComparator(MyCompare.class);
the following previously legal line will now provoke a compile error:
dbConfig.setBtreeComparator(compareInstance.getClass());
The compile error is puzzling and regrettably breaks compile time
compatibility with JE 3.2 and earlier, but comes about because in
Java, due to its type erasure based generics scheme, all the instances
of a generic class have the same runtime class. Instead, the
application has to apply the following cast:
dbConfig.setBtreeComparator((Class<? extends Comparator<byte[]>>) compareInstance.getClass());
Utility Changes:
-
Database salvage mode is now available in the API through the
com.sleepycat.je.util.DbScavenger class rather than solely from
the DbDump -r command line. This helped to fix and clarify a mismatch
between the Javadoc and the constructor parameters for DbDump in the
JE 3.2.X code line. See the com.sleepycat.je.util.DbScavenger class.
[#15385]
Configuration, Documentation, Test and Build Changes:
-
JE 3.3 requires Java 1.5.0.12 or higher.
-
Ant 1.7.0 is now required to build JE 3.3.
-
In the past, the javadoc for JE's public classes was generated through
an internal documentation process, and the actual content of the
comments was not in the source itself. This was inconvenient for IDE
users, because the javadoc contents would often not display in
convenient places. This has been changed, and the javadoc content now
resides in the source code in the normal fashion.
-
Java developers often approach JE with past experience using SQL. To
help illustrate the JE data model and API, we created a new example
program which takes a list of common SQL queries and implements them
using the Direct Persistent Layer API (DPL). Please refer to the
Translating SQL Queries Example in JE Installation Notes for more
information.
-
The example.properties file is no longer used to document the configuration
parameters that are used with EnvironmentConfig and EnvironmentMutableConfig.
Instead, String constants have been added EnvironmentConfig which include that
documentation. When calling setConfigParam these constants can be used to
avoid hard-coding the parameter name. See EnvironmentConfig for details.
[#16227]