Summary

The JRuby implementation of msgpack (msgpack gem, java platform — pure Java decoder under ext/java/) sizes containers directly from the element count written in the message header, with no cap at all: no equivalent of the C unpacker's SHRT_MAX clamp (since 1.7.3) and none of the buffered-bytes cap that commit c8d42d8 added for CRuby in 1.8.6. A single 5-byte message decides how much memory a JVM process tries to allocate:

• array32 header declaring 0x7FFFFFFF elements → new IRubyObject[0x7FFFFFFF] (~8 GiB of reference slots) attempted immediately → java.lang.OutOfMemoryError: Requested array size exceeds VM limit.

• Counts below the VM's max-array-size but above available heap (e.g. 5×10⁸ with a 512 MiB heap) produce a real allocation attempt — OutOfMemoryError: Java heap space — the JVM genuinely tries to take the memory before failing.

• str32/bin32 headers declaring 0x7FFFFFFF bytes do the same via readBytes(size) → new byte[size].

• Below the failure threshold there is a silent band: one streaming unpacker fed only an array32 header with count 2×10⁷ (5 wire bytes, zero elements) grows used heap +78 MiB immediately, no exception (each swallows the underflow), and the footprint stays elevated after Unpacker#reset + full GC. Linear in declared count (~4 B/element); nothing bounds it — not delivered bytes, not a per-container cap.

Only counts ≥ 2³¹ are refused (a clean MessagePack::UnderflowError from getUnsignedInt()), and only maps are safe (consumeHash grows per element instead of pre-sizing). Verified on the released gem msgpack 1.8.6 java, JRuby 9.4.12 / OpenJDK 25, 2026-10-30 — i.e. unfixed in the current release. Proposed fix mirrors c8d42d8: cap initial allocation by bytes actually buffered and grow as data arrives; read strings in bounded chunks instead of allocating the declared length up front.

Impact

Type. Uncontrolled resource consumption driven by declared sizes rather than delivered bytes — CWE-770 (Allocation of Resources Without Limits or Throttling), precisely CWE-1325 (Independent Resource Usage Limiting Not Applied). Availability only; no confidentiality/integrity impact, no code execution.

Who is impacted. Any JRuby application that feeds untrusted msgpack to MessagePack.unpack/Unpacker — the java-platform gem's whole user base (JRuby-based services, log/RPC pipelines, JVM polyglot stacks using this gem). For them: remote OOM of a shared-heap JVM from a 5-byte message, or silent heap spend below the exception band.