Saturday, July 27, 2024
Google search engine
HomeUncategorizedJEP Draft: Deprecate memory-access methods in sun.misc.unsafe for removal

JEP Draft: Deprecate memory-access methods in sun.misc.unsafe for removal

Summary

Deprecate the memory-access methods in sun.misc.Unsafe for removal in a future release. These unsupported methods have had supported replacements since JDK 9 (for accessing on-heap memory) and JDK 22 (for accessing off-heap memory). Library developers are strongly encouraged to migrate from sun.misc.Unsafe to the supported replacements, so that applications can migrate smoothly to modern JDKs.

Goals

  • Prepare for the removal of the memory-access methods in sun.misc.Unsafe in a future JDK release.

  • Help developers know when their applications rely, directly or indirectly, on the memory-access methods in sun.misc.Unsafe.

Non-Goals

  • It is not a goal to remove the sun.misc.Unsafe class entirely. A small number of its methods are not used for memory access and will remain undeprecated. Their deprecation and removal will be done separately.

  • It is not a goal to change other sun.* classes in the jdk.unsupported module.

Motivation

The sun.misc.Unsafe class was introduced in 2002 as a way for Java classes in the JDK to perform low-level operations. Most of its methods — 79 out of 87 — are for accessing memory, either in the JVM’s garbage-collected heap or in “off-heap” memory not controlled by the JVM. As the name of the class suggests, these memory-access methods are unsafe — they can lead to undefined behavior, including JVM crashes — so they were not exposed as a standard API. They were neither envisaged for use by a broad range of clients nor intended to be permanent. Rather, they were introduced with the assumption that they were exclusively for use within the JDK, and that callers within the JDK would perform exhaustive safety checks before using them, and that safe standard APIs for this functionality would eventually be added to the Java Platform.

However, with no way in 2002 to prevent sun.misc.Unsafe from being used outside the JDK, its memory-access methods became a “Swiss army knife” for library developers who wanted more power and performance than standard APIs could offer. For example, sun.misc.Unsafe::compareAndSwap can perform a CAS (compare-and-swap) operation on a field without the overhead of java.util.concurrent.atomic, while sun.misc.Unsafe::setMemory can manipulate off-heap memory without the 2GB limitation of java.nio.ByteBuffer.

Unfortunately, not all libraries are diligent at performing safety checks before calling the memory-access methods, so there is a risk of failures and crashes in applications. Also, many uses of the methods are unnecessary, driven by the ease of copy-and-paste from online forums. Nevertheless, because use of the memory-access methods is so widespread, sun.misc.Unsafe was not encapsulated alongside other low-level APIs in JDK 9. It remains available “out of the box” in JDK 22, pending the availability of safe supported alternatives.

Over the past several years, we have introduced two standard APIs that are safe and performant replacements for the memory-access methods in sun.misc.Unsafe:

  • java.lang.invoke.VarHandle, introduced in JDK 9, provides methods to safely and efficiently manipulate on-heap memory: fields of objects, static fields of classes, and elements of arrays.

  • java.lang.foreign.MemorySegment, introduced in JDK 22, provides methods to safely and efficiently access off-heap memory (sometimes in cooperation with VarHandle).

These standard APIs guarantee no undefined behavior, promise long-term stability, and have high-quality integration with the tooling and documentation of the Java Platform (examples of their use are given below). Given the availability of these APIs, it is now appropriate to deprecate and eventually remove the memory-access methods in sun.misc.Unsafe.

The removal of the memory-access methods in sun.misc.Unsafe is part of a coordinated effort to provide integrity by default in the Java Platform. Other initiatives include restrictions on JNI and on the dynamic loading of agents. These efforts are necessary to make the Java Platform more secure and performant. They also mean that application developers will no longer get stuck on older JDKs because of libraries that break on newer JDKs when unsupported APIs are changed.

Description

The memory-access methods of sun.misc.Unsafe can be divided into three categories:

  • Methods for accessing on-heap memory (on-heap),
  • Methods for accessing off-heap memory (off-heap), and
  • Methods for accessing both on-heap and off-heap memory (bimodal — a bimodal method takes a parameter which either refers to an object on the heap or is null to signify off-heap access).

We will proceed as follows, where each phase takes place in a separate successive JDK feature release:

  1. Deprecate all memory-access methods — on-heap, off-heap, and bimodal — for removal. This will cause compile-time deprecation warnings for code that refers to the methods, alerting library developers to their forthcoming removal. A new command-line option, described below, will allow users to receive runtime warnings when the methods are used.

    Distinct from deprecation warnings, javac has given warnings about the use of sun.misc.Unsafe since 2006:

    warning: Unsafe is internal proprietary API and may be removed in a future release

    These warnings will continue to appear, and cannot be suppressed.

  2. Issue a warning at run time, as detailed below, when a memory-access method is used (including through reflection). This will alert application developers and users to the forthcoming removal of the methods, and the need to upgrade libraries in future.

  3. Throw an exception when a memory-access method is used (including through reflection). This will further alert application developers and users to the imminent removal of the methods.

  4. Remove on-heap methods. These methods will be removed first because they have had standard replacements since JDK 9 in 2017.

  5. Remove off-heap and bimodal methods. These methods will be removed later because they have had standard replacements only since JDK 22 in 2023.

We plan to implement the run-time warnings of phase 2 in or before JDK 25.

We plan to implement the removal of methods in phases 4 and 5 in JDK 26 or later. We may perform phases 4 and 5 simultaneously if appropriate when the time arrives.

Allowing the use of memory-access methods in sun.misc.Unsafe

The vast majority of Java developers do not use sun.misc.Unsafe explicitly in their own code. However, many applications depend, directly or indirectly, on libraries that use the memory-access methods of sun.misc.Unsafe. Starting in JDK 23, application developers can assess how the methods’ deprecation and removal will affect libraries on the class path by running with a new command line option, --sun-misc-unsafe-memory-access={allow|warn|debug|deny} (which is similar, in spirit and in form, to the --illegal-access flag introduced by JEP 261):

  • --sun-misc-unsafe-memory-access=allow allows use of the memory-access methods with no warnings at run time.

  • --sun-misc-unsafe-memory-access=warn allows use of the memory-access methods, but issues a warning on the first occasion that any memory-access method is used (including through reflection). That is, there is at most one warning regardless of which memory-access methods are used and how many times any particular method is used.

  • --sun-misc-unsafe-memory-access=debug allows use of the memory-access methods, but issues a one-line warning and a stack trace on every occasion that a memory-access method is used (including through reflection).

  • --sun-misc-unsafe-memory-access=deny disallows use of the memory-access methods by throwing an UnsupportedOperationException on every occasion that any such method is used (including through reflection).

The warning from warn is as follows:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::setMemory has been called by com.foo.bar.Server (file:/tmp/foobarserver/thing.jar)
WARNING: Please consider reporting this to the maintainers of com.foo.bar.Server
WARNING: sun.misc.Unsafe::setMemory will be removed in a future release

The default value of --sun-misc-unsafe-memory-access will change from release to release as we proceed with the deprecation and removal of memory-access methods:

  • allow is the default value in phase 1, as if every invocation of the java launcher includes --sun-misc-unsafe-memory-access=allow.

  • warn will be the default value in phase 2. It will be possible in phase 2 to “dial back” the value from warn to allow, thus avoiding a warning.

  • deny will be the default value in phase 3. It will be possible in phase 3 to “dial back” the value from deny to warn in order to receive a warning rather than exceptions. It will not be possible to use allow to avoid a warning entirely.

  • In phase 5, when all the memory-access methods have been removed, --sun-misc-unsafe-memory-access will be ignored. Eventually it will be removed.

The following tools in the JDK help advanced developers to understand how their application or library uses deprecated methods in sun.misc.Unsafe:

  • javac gives a deprecation warning whenever source code uses a memory-access method in sun.misc.Unsafe. Each deprecation warning can be suppressed in code with @SuppressWarnings("removal").

  • jdeprscan is a tool that reports uses of deprecated APIs by classes in a JAR file. It can detect where memory-access methods of sun.misc.Unsafe are used by an application or any of its dependencies.

  • JDK Flight Recorder (JFR): When JFR is enabled, the jdk.DeprecatedInvocation event will be emitted whenever a terminally deprecated method of sun.misc.Unsafe is called. This event was added in JDK 22.

sun.misc.Unsafe memory-access methods and their replacements

On-heap methods

  • long objectFieldOffset(Field f)
  • long staticFieldOffset(Field f)
  • Object staticFieldBase(Field f)
  • int arrayBaseOffset(Class arrayClass)
  • int arrayIndexScale(Class arrayClass)

These methods exist to obtain offsets (or scale) which are then used with the bimodal methods below to read and write fields or array elements. This usage is replaced by VarHandle and MemorySegment::ofArray.

In rare circumstances, these methods are used on their own to examine and manipulate the physical layout of an object in memory (see examples here). There is no supported replacement for this use case; see below for further discussion.

The first three methods above were already deprecated in JDK 18.

In addition, the following fields that are associated with these methods are deprecated for removal:

  • int INVALID_FIELD_OFFSET
  • int ARRAY_[TYPE]_BASE_OFFSET
  • int ARRAY_[TYPE]_INDEX_SCALE

Off-heap methods

FFM downcalls, referenced below, are described here.

Bimodal memory-access methods

Migration examples

On-heap memory access

Suppose class Foo has an int field that we wish to atomically double. With sun.misc.Unsafe, the code is, assuming UNSAFE is a static field that refers to a sun.misc.Unsafe object:

class Foo {
    private int x;
    
    public boolean tryAtomicallyDoubleX() {
        int oldValue = x;
        return UNSAFE.compareAndSwapInt(this, X_OFFSET, oldValue, oldValue*2);
    }
    
    private static final long X_OFFSET;
    static {
        try {
            X_OFFSET = UNSAFE.objectFieldOffset(Foo.class.getDeclaredField("x"));
        } catch (Exception ex) { throw new AssertionError(ex); }
    }
}

With the standard VarHandle API, the code is:

class Foo {
    private int x;
    
    public boolean tryAtomicallyDoubleX() {
        int oldValue = x;
        return X_VH.compareAndSet(this, oldValue, oldValue*2);
    }
    
    private static final VarHandle X_VH;
    static {
        try {
            X_VH = MethodHandles.lookup().findVarHandle(Foo.class, "x", int.class);
        } catch (Exception ex) { throw new AssertionError(ex); }
    }
}

Here is a class that uses sun.misc.Unsafe to perform a volatile write of an array element:

class Foo {
    private int[] a = new int[10];
    
    public void setVolatile(int index, int value) {
        if (index < 0 || index >= a.length)
            throw new ArrayIndexOutOfBoundsException(index);
        UNSAFE.putIntVolatile(a, ARRAY_BASE + ARRAY_SCALE*index, value);
    }
    
    private static final int ARRAY_BASE = UNSAFE.arrayBaseOffset(int[].class);
    private static final int ARRAY_SCALE = UNSAFE.arrayIndexScale(int[].class);
}

With VarHandle, the code is:

class Foo {
    private int[] a = new int[10];
    
    public void setVolatile(int index, int value) {
        AVH.setVolatile(a, index, value);
    }
    
    private static final VarHandle AVH = MethodHandles.arrayElementVarHandle(int[].class);
}

Off-heap memory access

Here is a class that uses sun.misc.Unsafe to allocate an off-heap buffer and perform three operations: a volatile write of an int, a bulk initialization of a subset of the buffer, and a copy of the buffer data into a Java int array:

class OffHeapIntBuffer {
    private final long size;
    private long bufferPtr;
    
    public OffHeapIntBuffer(long size) {
        this.size = size;
        this.bufferPtr = UNSAFE.allocateMemory(size*ARRAY_SCALE);
    }
    
    public void deallocate() {
        if (bufferPtr == 0) return;
        UNSAFE.freeMemory(bufferPtr);
        bufferPtr = 0;
    }
    
    public void setVolatile(long index, int value) {
        checkBounds(index);
        UNSAFE.putIntVolatile(null, bufferPtr + ARRAY_SCALE*index, value);
    }
    
    public void initialize(long start, long n) {
        checkBounds(start);
        checkBounds(start + n-1);
        UNSAFE.setMemory(bufferPtr + start*ARRAY_SCALE, n*ARRAY_SCALE, 0);
    }
    
    public int[] copyToNewArray(long start, int n) {
        checkBounds(start);
        checkBounds(start + n-1);
        int[] a = new int[n];
        UNSAFE.copyMemory(null, bufferPtr + start*ARRAY_SCALE, a, ARRAY_BASE, n*ARRAY_SCALE);
        return a;
    }
    
    private boolean checkBounds(long index) {
        if (index < 0 || index >= size)
            throw new IndexOutOfBoundsException(index);
        return true;
    }
    
    private static final int ARRAY_BASE = UNSAFE.arrayBaseOffset(int[].class);
    private static final int ARRAY_SCALE = UNSAFE.arrayIndexScale(int[].class);
}

With the standard Arena and MemorySegment APIs, the code is:

class OffHeapIntBuffer {
    private static final VarHandle ELEM_VH = ValueLayout.JAVA_INT.arrayElementVarHandle();

    private final Arena arena;
    private final MemorySegment buffer;
    
    public OffHeapIntBuffer(long size) {
        this.arena  = Arena.ofShared();
        this.buffer = arena.allocate(ValueLayout.JAVA_INT, size);
    }
    
    public void deallocate() {
        arena.close();
    }
    
    public void setVolatile(long index, int value) {
        ELEM_VH.setVolatile(buffer, 0L, index, value);
    }
    
    public void initialize(long start, long n) {
        buffer.asSlice(ValueLayout.JAVA_INT.byteSize() * start,
                       ValueLayout.JAVA_INT.byteSize() * n)
              .fill((byte) 0);
    }
    
    public int[] copyToNewArray(long start, int n) {
        return buffer.asSlice(ValueLayout.JAVA_INT.byteSize() * start,
                              ValueLayout.JAVA_INT.byteSize() * n)
                     .toArray(ValueLayout.JAVA_INT);
    }
}

Risks and Assumptions

  • Over the years, methods in sun.misc.Unsafe that are unrelated to memory access have been deprecated for removal after standard replacements were introduced, and many of them have already been removed:

    • sun.misc.Unsafe::defineClass was removed in JDK 11 after java.lang.invoke.MethodHandles.Lookup::defineClass was introduced in JDK 9.

    • sun.misc.Unsafe::defineAnonymousClass was removed in JDK 17 after MethodHandles.Lookup::defineHiddenClass was introduced in in JDK 15.

    • sun.misc.Unsafe::{ensureClass,shouldBe}Initialized were removed in JDK 22 after MethodHandles.Lookup::ensureInitialized was introduced in JDK 15.

    • Six methods in sun.misc.Unsafe were deprecated for removal in JDK 22 after the availability of standard replacements.

    We have seen very little impact in the Java ecosystem from the removal of these relatively obscure methods. However, the memory-access methods are much better known. This proposal assumes that removing them will impact libraries. Accordingly, for maximum visibility, we are proposing their deprecation and removal via the JEP process rather than just with a CSR request and a release note.

  • This proposal assumes that library developers will migrate from unsupported methods in sun.misc.Unsafe to supported methods in java.*.

    We recommend in the strongest possible terms that library developers do not migrate from unsupported methods in sun.misc.Unsafe to unsupported methods found elsewhere inside the JDK.

    Library developers who ignore this recommendation will force their users to run with --add-exports or --add-opens options on the command line. This is not merely inconvenient; it is risky because JDK internals can change from release to release without notice, breaking libraries which depend on the internals, and breaking applications which depend on the libraries.

  • A risk of this proposal is that some libraries use the on-heap memory-access methods of sun.misc.Unsafe in ways that cannot be replicated by the standard APIs available in JDK 23. For example, a library may use Unsafe::objectFieldOffset to obtain a field’s offset in an object, then use Unsafe::putInt to write an int value at that offset regardless of whether the field is an int. The standard VarHandle API cannot examine or manipulate objects at such a low level because it refers to fields by name and type, not by offset. Use cases that rely on knowledge of field offsets are, in effect, revealing or exploiting implementation details of the JVM. In our view, such use cases do not need to be supported by a standard API.

  • A library may use Unsafe.getInt(array, arrayBase + offset) to access array elements in the heap without any bounds checking. This idiom is typically used for random access, because sequential (counted) access to array elements via both the MemorySegment API and ordinary array[index] code already benefits from bound-check elimination. Random access to array elements without bounds checking is not a use case that we feel needs to be preserved in a supported API. Random access via the MemorySegment API or array[index] has a small loss of performance compared to the on-heap memory access methods of sun.misc.Unsafe, but a large gain in safety and maintainability. In particular, the use of standard APIs is guaranteed to work reliably on all platforms and JDK releases, even if the JVM’s implementation of arrays is changed in future.

Future Work

After deprecating the 79 memory-access methods for removal, sun.misc.Unsafe will contain only three methods that are not deprecated:

  • pageSize, which will be deprecated and removed separately. Library developers are encouraged to obtain page size directly from the OS via a downcall.

  • throwException, which will be deprecated and removed separately. This method was historically used by methods in the JDK to wrap checked exceptions in unchecked exceptions, but those methods, e.g., Class::newInstance, are now deprecated.

  • allocateInstance, which will remain as the only method in sun.misc.Unsafe in the medium term. It is used by some serialization libraries for deserialization, and providing a standard replacement is a long-term project.

Read More

RELATED ARTICLES

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here

- Advertisment -
Google search engine

Most Popular

Recent Comments