SIP-73 - Adding Flexible Types as Internal Type to Scala 3 Spec

content/flexible-types.md

@@ -14,7 +14,7 @@ title: SIP-XX - Flexible Types for TASTy Format
 
 ## Summary
 
-This proposal specifies the encoding of Flexible Types in the TASTy (Typed Abstract Syntax Tree) format. Flexible Types are a compiler-internal mechanism introduced to improve Java and legacy Scala code (compiled without explicit nulls) interoperability under explicit nulls (`-Yexplicit-nulls`). They allow reference types from Java libraries and legacy Scala codeto be treated as either nullable or non-nullable depending on the usage.
+This proposal specifies the encoding of Flexible Types in the TASTy (Typed Abstract Syntax Tree) format. Flexible Types are an Internal Type (see §3.1 of the Scala 3 language specification) introduced to improve interoperability with Java and legacy Scala code (compiled without explicit nulls) under explicit nulls (`-Yexplicit-nulls`). They allow reference types from Java libraries and legacy Scala code to be treated as either nullable or non-nullable depending on the usage.
 
 Flexible Types provide a type-safe bridge between implicit nullability and Scala's explicit null system, enabling smoother interoperation while maintaining safety guarantees where possible. This SIP formalizes their representation in TASTy to ensure consistent serialization and deserialization across compiler versions.
 
@@ -48,61 +48,75 @@ Both approaches are suboptimal for large codebases that want gradual migration t
 
 ### High-level overview
 
-We introduce Flexible Types as a compiler-internal representation that allows a type to be treated as both nullable and non-nullable depending on the context. A Flexible Type `T?` (notation borrowed from Kotlin's platform types) has bounds `T | Null <: T? <: T`, meaning:
+We introduce Flexible Types as an Internal Type that allows a type to be treated as both nullable and non-nullable depending on the context. Informally, we write a flexible type as `T?` (notation inspired by Kotlin platform types). The following subtyping relationships hold: `T | Null <: T?` and `T? <: T`, meaning:
 
 - It can accept both `T` and `T | Null` values
 - It can be used where either `T` or `T | Null` is expected
-- It can be called with member functions of `T`, but may throw `NullPointerException` at runtime if the value is actually null
+- It can be used as the prefix in accesses to members of `T`, but may throw `NullPointerException` at runtime if the value is actually null
 
 Flexible Types are **non-denotable** - users cannot write them explicitly in source code. Only the compiler creates them during Java interoperability and when consuming legacy Scala code.
 
 They may appear in type signatures because of type inference.
 Due to their non-denotable nature, we do not recommend exposing Flexible Types in public APIs or library interfaces.
-We will implement a mechanism to warn users when Flexible Types are exposed at field or method boundaries.
+We have implemented a mechanism to warn users when Flexible Types are exposed at public (or protected) field or method boundaries.
 
 ### Specification
 
-#### TASTy Format Extension
+#### Abstract Syntax (Spec Addendum)
 
-We extend the TASTy format with a new type tag (`193`):
+We extend the abstract syntax of (internal) types with a new form:
 
 ```
-FLEXIBLEtype   Length underlying_Type                            -- (underlying)?
+InternalType ::= ... | FlexibleType
+FlexibleType ::= Type ‘?’
 ```
 
-The tag is followed by the length of the underlying type and the underlying type itself.
+`T?` (rendered informally in spec; there is no concrete syntax) designates a flexible type whose underlying type is `T`.
+
+Normalization: `(T?)? = T?` (flexible types do not nest).
+
+#### Conformance (Extension to §3.6.1)
+
+We extend the conformance relation (<:) with the following two derivation rules:
+
+1. `S = U` and `T = U?`
+2. `S = Null` and `T = U?`
+3. `S = U?` and `T = U`
+
+We can also equivalence: `U =:= U?` and `U | Null =:= U?`, 
+even though `U | Null` and `U` may be not equivalent under explicit nulls.
 
-The underlying type `T` represents the upper bound of the flexible type, and the lower bound is implicitly `T | Null`.
+#### Member Selection
+
+Member selection is treated as if `T?` were `T` (so `memberType(T?, m, p)` delegates to `memberType(T, m, p)`).
+
+#### TASTy Format Extension
 
-#### Subtyping Rules
+We reserve a new TASTy type tag (`193`) to encode flexible types:
 
-Flexible Types are designed to introduce a controlled soundness hole to enable practical interoperability. Their subtyping rules differ from regular types:
+```
+FLEXIBLEtype   Length underlying_Type
+```
 
-The subtyping relationships for Flexible Type `T?` are:
+Decoders that do not recognize `FLEXIBLEtype` may safely treat it as its underlying type `T` (erasure compatibility is preserved).
 
-1. **Lower bound**: `T | Null <: T?`
-2. **Upper bound**: `T? <: T`
+#### Subtyping Rules in Compiler
 
-Implementation in `TypeComparer.scala`:
+For implementors: the two conformance rules above are implemented in `TypeComparer.scala` as follows:
 
 ```scala
-// In firstTry method (line ~901)
+// In firstTry method 
 case tp2: FlexibleType =>
   recur(tp1, tp2.lo)  // tp1 <: FlexibleType.lo (which is T | Null)
 
-// In thirdTry method (line ~1098)
+// In thirdTry method
 case tp1: FlexibleType =>
   recur(tp1.hi, tp2)  // FlexibleType.hi (which is T) <: tp2
 ```
 
-#### Member Selection
-
-All members of the underlying type `T` are considered to be members of the flexible type `T?`.
-Selecting a member from `T?` may throw `NullPointerException` at runtime if the actual value is `null`.
-
-#### Erasure
+#### Type Erasure (Extension to §3.8)
 
-The erased type of `T?` is the erased type of the underlying type `T`.
+Erasure is extended with: `|T?| = |T|` (i.e., identical to the erasure of its underlying type).
 
 ### Compatibility
 
@@ -118,42 +132,19 @@ Flexible Types have already been implemented in the latest Scala 3 compiler. The
 
 ### Core Implementation Status
 
-1. **Type System Integration**: The `FlexibleType` case class and its core subtyping rules have been implemented in `Types.scala`
-2. **Subtyping Logic**: The subtyping algorithms in `TypeComparer.scala` handle flexible types according to the specification
-3. **TASTy Serialization**: The `FLEXIBLEtype` tag (`193`) is fully implemented in `TastyFormat.scala` and supports serialization/deserialization
-4. **Nullification Rules**: Both Java classes and legacy Scala code are processed with flexible type nullification when `-Yexplicit-nulls` is enabled
+1. **New Type and Subtyping Rules**: The `FlexibleType` case class and its subtyping rules have been implemented
+2. **TASTy Serialization**: The `FLEXIBLEtype` tag (`193`) is fully implemented in `TastyFormat.scala` and supports serialization/deserialization
+3. **Nullification Rules**: Both Java classes and legacy Scala code are processed with flexible type nullification when `-Yexplicit-nulls` is enabled
+4. **Public API Warnings**: A warning mechanism is in place to alert users when flexible types appear in public or protected API boundaries
 
 ### Planned Improvements
 
 The following enhancements are planned for upcoming releases:
 
 1. Refined nullification rules for edge cases.
 2. Stronger TASTy forward/backward compatibility guarantees, including updating tasty-mima and tasty-query.
-3. A compiler warning when flexible types appear in public API boundaries.
 
 ## Related information
 
 - [**Explicit Nulls**](https://docs.scala-lang.org/scala3/reference/experimental/explicit-nulls.html): The experimental explicit nulls feature that motivated the need for flexible types.
-- [**Kotlin Platform Types**](https://kotlinlang.org/docs/java-interop.html#null-safety-and-platform-types): Direct inspiration for the flexible types concept, providing similar interoperability between Kotlin's null safety and Java's implicit nullability.
-
-## FAQ
-
-### Why are Flexible Types not user-denotable?
-
-Making flexible types non-denotable prevents users from depending on them in API boundaries. This ensures:
-
-1. APIs remain clean and explicit about nullability
-2. Flexible types serve only as an interop mechanism, not a permanent type system feature
-3. Migration path remains clear - eventually all types should be either `T` or `T | Null`
-
-### Can Flexible Types be nested?
-
-No, flexible types cannot be nested. `FlexibleType(FlexibleType(T))` is normalized to `FlexibleType(T)`.
-This prevents unnecessarily complex type representations.
-
-### What happens with generic wildcards?
-
-Generic wildcards are handled specially:
-- `List<?>` becomes `List[?]?` (flexible type with a wildcard bounds)
-- `List<? extends String>` becomes `List[? <: String?]?`
-- The outer container is made flexible to handle implicit nullability
+- [**Kotlin Platform Types**](https://kotlinlang.org/docs/java-interop.html#null-safety-and-platform-types): Direct inspiration for the flexible types concept, providing similar interoperability between Kotlin's null safety and Java's implicit nullability.