This eliminates the statically-reachable recursion loop between code
generation backends and Sema. This is beneficial for optimizers
(although I do not measure any performance improvement for this change),
and for profilers.
Note that the `_ = Address` statements in tests previously were a nop,
and now actually check that the type is valid. However, on WASI, the
type is *not* valid.
I'm so sorry.
This commit was just meant to be making all types fully resolve by
queueing resolution at the moment of their creation. Unfortunately, a
lot of dominoes ended up falling. Here's what happened:
* I added a work queue job to fully resolve a type.
* I realised that from here we could eliminate `Sema.types_to_resolve`
if we made function codegen a separate job. This is desirable for
simplicity of both spec and implementation.
* This led to a new AIR traversal to detect whether any required type is
unresolved. If a type in the AIR failed to resolve, then we can't run
codegen.
* Because full type resolution now occurs by the work queue job, a bug
was exposed whereby error messages for type resolution were associated
with the wrong `Decl`, resulting in duplicate error messages when the
type was also resolved "by" its owner `Decl` (which really *all*
resolution should be done on).
* A correct fix for this requires using a different `Sema` when
performing type resolution: we need a `Sema` owned by the type. Also
note that this fix is necessary for incremental compilation.
* This means a whole bunch of functions no longer need to take `Sema`s.
* First-order effects: `resolveTypeFields`, `resolveTypeLayout`, etc
* Second-order effects: `Type.abiAlignmentAdvanced`, `Value.orderAgainstZeroAdvanced`, etc
The end result of this is, in short, a more correct compiler and a
simpler language specification. This regressed a few error notes in the
test cases, but nothing that seems worth blocking this change.
Oh, also, I ripped out the old code in `test/src/Cases.zig` which
introduced a dependency on `Compilation`. This dependency was
problematic at best, and this code has been unused for a while. When we
re-enable incremental test cases, we must rewrite their executor to use
the compiler server protocol.
This change modifies `Zcu.ErrorMsg` to store a `Zcu.LazySrcLoc` rather
than a `Zcu.SrcLoc`. Everything else is dominoes.
The reason for this change is incremental compilation. If a failed
`AnalUnit` is up-to-date on an update, we want to re-use the old error
messages. However, the file containing the error location may have been
modified, and `SrcLoc` cannot survive such a modification. `LazySrcLoc`
is designed to be correct across incremental updates. Therefore, we
defer source location resolution until `Compilation` gathers the compile
errors into the `ErrorBundle`.
Previously, `reference_table` mapped from a `Decl` being referenced to
the `Decl` that performed the reference. This is convenient for
constructing error messages, but problematic for incremental
compilation. This is because on an incremental update, we want to
efficiently remove all references triggered by an `AnalUnit` which is
being re-analyzed.
For this reason, `reference_table` now maps the other way: from the
`AnalUnit` *performing* the reference, to the `AnalUnit` whose analysis
was triggered. As a general rule, any call to any of the following
functions should be preceded by a call to `Sema.addReferenceEntry`:
* `Zcu.ensureDeclAnalyzed`
* `Sema.ensureDeclAnalyzed`
* `Zcu.ensureFuncBodyAnalyzed`
* `Zcu.ensureFuncBodyAnalysisQueued`
This is not just important for error messages, but also more
fundamentally for incremental compilation. When an incremental update
occurs, we must determine whether any `AnalUnit` has become
unreferenced: in this case, we should ignore its associated error
messages, and perhaps even remove it from the binary. For this reason,
we no longer store only one reference to every `AnalUnit`, but every
reference. At the end of an update, `Zcu.resolveReferences` will
construct the reverse mapping, and as such identify which `AnalUnit`s
are still referenced. The current implementation doesn't quite do what
we need for incremental compilation here, but the framework is in place.
Note that `Zcu.resolveReferences` does constitute a non-trivial amount
of work on every incremental update. However, for incremental
compilation, this work -- which will effectively be a graph traversal
over all `AnalUnit` references -- seems strictly necessary. At the
moment, this work is only done if the `Zcu` has any errors, when
collecting them into the final `ErrorBundle`.
An unsolved problem here is how to represent inline function calls in
the reference trace. If `foo` performs an inline call to `bar` which
references `qux`, then ideally, `bar` would be shown on the reference
trace between `foo` and `qux`, but this is not currently the case. The
solution here is probably for `Zcu.Reference` to store information about
the source locations of active inline calls betweeen the referencer and
its reference.
This change seeks to more appropriately model the way semantic analysis
works by drawing a more clear line between errors emitted by analyzing a
`Decl` (in future a `Cau`) and errors emitted by analyzing a runtime
function.
This does change a few compile errors surrounding compile logs by adding
more "also here" notes. The new notes are more technically correct, but
perhaps not so helpful. They're not doing enough harm for me to put
extensive thought into this for now.
This commit reworks our representation of exported Decls and values in
Zcu to be memory-optimized and trivially serialized.
All exports are now stored in the `all_exports` array on `Zcu`. An
`AnalUnit` which performs an export (either through an `export`
annotation or by containing an analyzed `@export`) gains an entry into
`single_exports` if it performs only one export, or `multi_exports` if
it performs multiple.
We no longer store a persistent mapping from a `Decl`/value to all
exports of that entity; this state is not necessary for the majority of
the pipeline. Instead, we construct it in `Zcu.processExports`, just
before flush. This does not affect the algorithmic complexity of
`processExports`, since this function already iterates all exports in
the `Zcu`.
The elimination of `decl_exports` and `value_exports` led to a few
non-trivial backend changes. The LLVM backend has been wrangled into a
more reasonable state in general regarding exports and externs. The C
backend is currently disabled in this commit, because its support for
`export` was quite broken, and that was exposed by this work -- I'm
hoping @jacobly0 will be able to pick this up!
This is essentially just a rename. I also changed the representation of
`AnalSubject` to use a `packed struct` rather than a non-exhaustive
enum, but that change is relatively trivial.
LLVM fails to notice that in release builds, `logFn` ignores its
arguments, so their computation can be elided. So, LLVM fails to elide
this hashmap lookup. Its cost isn't too significant, but doing it in the
hottest loop in Sema adds up!
Technically, we could do the lookup a single time, before the loop, but
it was cleanest (and a little faster) to just disable this log call at
comptime when debug logging is disabled.
Since we track `reify` instructions across incremental updates, it is
acceptable to treat it as the baseline for a relative source location.
This turns out to be a good idea, since it makes it easy to define the
source location for a reified type.
When analyzing `zirFunc` to instantiate a generic function,
`sema.owner_decl` is not the owner Decl of the generic instance, but
instead of the call site, so that dependencies are propagated correctly.
(This aligns with the fact that in future, generic instantiations will
not have a corresponding `Cau`.) So, when deciding the callconv in this
case, we must check `sema.generic_owner` to determine whether the
function is exported.
🦀 src_decl is gone 🦀
This commit eliminates the `src_decl` field from `Sema.Block`. This
change goes further to eliminating unnecessary responsibilities of
`Decl` in preparation for its major upcoming refactor.
The two main remaining reponsibilities had to do with namespace types:
`src_decl` was used to determine their line number and their name. The
former use case is solved by storing the line number alongside type
declarations (and reifications) in ZIR; this is actually more correct,
since previously the line number assigned to the type was really the
line number of the source declaration it was syntactically contained
within, which does not necessarily line up. Consequently, this change
makes debug info for namespace types more correct, although I am not
sure how debuggers actually utilize this line number, if at all. Naming
types was solved by a new field on `Block`, called `type_name_ctx`. In a
sense, it represents the "namespace" we are currently within, including
comptime function calls etc. We might want to revisit this in future,
since the type naming rules seem to be a bit hand-wavey right now.
As far as I can tell, there isn't any more preliminary work needed for
me to start work on the behemoth task of splitting `Zcu.Decl` into the
new `Nav` (Named Addressable Value) and `Cau` (Comptime Analysis Unit)
types. This will be a sweeping change, impacting essentially every part
of the pipeline after `AstGen`.
`LazySrcLoc` now stores a reference to the "base AST node" to which it
is relative. The previous tagged union is `LazySrcLoc.Offset`. To make
working with this structure convenient, `Sema.Block` contains a
convenience `src` method which takes an `Offset` and returns a
`LazySrcLoc`.
The "base node" of a source location is no longer given by a `Decl`, but
rather a `TrackedInst` representing either a `declaration`,
`struct_decl`, `union_decl`, `enum_decl`, or `opaque_decl`. This is a
more appropriate model, and removes an unnecessary responsibility from
`Decl` in preparation for the upcoming refactor which will split it into
`Nav` and `Cau`.
As a part of these `Decl` reworks, the `src_node` field is eliminated.
This change aids incremental compilation, and simplifies `Decl`. In some
cases -- particularly in backends -- the source location of a
declaration is desired. This was previously `Decl.srcLoc` and worked for
any `Decl`. Now, it is `Decl.navSrcLoc` in reference to the upcoming
refactor, since the set of `Decl`s this works for precisely corresponds
to what will in future become a `Nav` -- that is, source-level
declarations and generic function instantiations, but *not* type owner
Decls.
This commit introduces more tags to `LazySrcLoc.Offset` so as to
eliminate the concept of `error.NeededSourceLocation`. Now, `.unneeded`
should only be used to assert that an error path is unreachable. In the
future, uses of `.unneeded` can probably be replaced with `undefined`.
The `src_decl` field of `Sema.Block` no longer has a role in type
resolution. Its main remaining purpose is to handle namespacing of type
names. It will be eliminated entirely in a future commit to remove
another undue responsibility from `Decl`.
It is worth noting that in future, the `Zcu.SrcLoc` type should probably
be eliminated entirely in favour of storing `Zcu.LazySrcLoc` values.
This is because `Zcu.SrcLoc` is not valid across incremental updates,
and we want to be able to reuse error messages from previous updates
even if the source file in question changed. The error reporting logic
should instead simply resolve the location from the `LazySrcLoc` on the
fly.
This is in preparation for some upcoming changes to how we represent
source locations in the compiler. The bulk of the change here is dealing
with the removal of `src()` methods from `Zir` types.
The justification for using relative source nodes in ZIR is that it
allows source locations -- which may be serialized across incremental
updates -- to be relative to the source location of their containing
declaration. However, having those "baseline" instructions themselves be
relative to their own parent is counterproductive, since the source
location updating problem is only being moved to `Decl`. Storing the
absolute node here instead makes more sense, since it allows for this
source location update logic to be elided entirely in the future by
storing a `TrackedInst.Index` to resolve a source location relative to
rather than a `Decl.Index`.
These instructions are not emitted by AstGen. They also would have no
effect even if they did appear in ZIR: the Sema handling for these
instructions creates a Decl which the name strategy is applied to, and
proceeds to never use it. This pointless CPU heater is now gone, saving
2 ZIR tags in the process.
* correctly report time spent analyzing function bodies
* print fully qualified decl names
* also have a progress node for codegen
The downside of these changes is that it's a bit flickerey, but the
upside is that it's accurate; you can see what the compiler's doing!
This fix is already in master branch for stdin, stdout, and stderr; this
commit solves the same problem but for the progress pipe.
Both fixes were originally included in one commit on this branch,
however it was split it into two so that master branch could receive the
fix before the progress branch is merged.
The added comment explains the issue here relatively well. The new
progress API made this bug obvious because it became visibly clear that
certain Compile steps were seemingly "hanging" until other steps
completed. As it turned out, these child processes had raced to spawn,
and hence one had inherited the other's stdio pipes, meaning the `poll`
call in `std.Build.Step.evalZigProcess` was not identifying the child
stdout as closed until an unrelated process terminated.
This was a "fake" type used to handle C varargs parameters, much like
generic poison. In fact, it is treated identically to generic poison in
all cases other than one (the final coercion of a call argument), which
is trivially special-cased. Thus, it makes sense to remove this special
tag and instead use `generic_poison_type` in its place. This fixes
several bugs in Sema related to missing handling of this tag.
Resolves: #19781
Certain types (notably, `std.ComptimeStringMap`) were resulting in excessively
long type names when instantiated, which in turn resulted in excessively long
symbol names. These are problematic for two reasons:
* Symbol names are sometimes read by humans -- they ought to be readable.
* Some other applications (looking at you, xcode) trip on very long symbol names.
To work around this for now, we cap the depth of value printing at 1, as opposed
to the normal 3. This doesn't guarantee anything -- there could still be, for
instance, an incredibly long aggregate -- but it works around the issue in
practice for the time being.
The operation `undefined & 0` ought to result in the value `0`, and likewise for
zeroing only some bits. `std/packed_int_array.zig` tests were failing because
this behavior was not implemented -- this issue was previously masked by faulty
bitcast logic which turned `undefined` values into `0xAA` on pointer loads.
Ideally, we would like to be able to track the undefined bits at comptime.
This is related to #19634.
This commit reverts the handling of partially-undefined values in
bitcasting to transform these bits into an arbitrary numeric value,
like happens on `master` today.
As @andrewrk rightly points out, #19634 has unfortunate consequences
for the standard library, and likely requires more thought. To avoid
a major breaking change, it has been decided to revert this design
decision for now, and make a more informed decision further down the
line.
I have noted several latent bugs in the handling of packed memory on
big-endian targets, and one of them has been exposed by the previous
commit, triggering this test failure. I am opting to disable it for now
on the ground that the commit preceding this one helps far more than it
hurts.
We've got a big one here! This commit reworks how we represent pointers
in the InternPool, and rewrites the logic for loading and storing from
them at comptime.
Firstly, the pointer representation. Previously, pointers were
represented in a highly structured manner: pointers to fields, array
elements, etc, were explicitly represented. This works well for simple
cases, but is quite difficult to handle in the cases of unusual
reinterpretations, pointer casts, offsets, etc. Therefore, pointers are
now represented in a more "flat" manner. For types without well-defined
layouts -- such as comptime-only types, automatic-layout aggregates, and
so on -- we still use this "hierarchical" structure. However, for types
with well-defined layouts, we use a byte offset associated with the
pointer. This allows the comptime pointer access logic to deal with
reinterpreted pointers far more gracefully, because the "base address"
of a pointer -- for instance a `field` -- is a single value which
pointer accesses cannot exceed since the parent has undefined layout.
This strategy is also more useful to most backends -- see the updated
logic in `codegen.zig` and `codegen/llvm.zig`. For backends which do
prefer a chain of field and elements accesses for lowering pointer
values, such as SPIR-V, there is a helpful function in `Value` which
creates a strategy to derive a pointer value using ideally only field
and element accesses. This is actually more correct than the previous
logic, since it correctly handles pointer casts which, after the dust
has settled, end up referring exactly to an aggregate field or array
element.
In terms of the pointer access code, it has been rewritten from the
ground up. The old logic had become rather a mess of special cases being
added whenever bugs were hit, and was still riddled with bugs. The new
logic was written to handle the "difficult" cases correctly, the most
notable of which is restructuring of a comptime-only array (for
instance, converting a `[3][2]comptime_int` to a `[2][3]comptime_int`.
Currently, the logic for loading and storing work somewhat differently,
but a future change will likely improve the loading logic to bring it
more in line with the store strategy. As far as I can tell, the rewrite
has fixed all bugs exposed by #19414.
As a part of this, the comptime bitcast logic has also been rewritten.
Previously, bitcasts simply worked by serializing the entire value into
an in-memory buffer, then deserializing it. This strategy has two key
weaknesses: pointers, and undefined values. Representations of these
values at comptime cannot be easily serialized/deserialized whilst
preserving data, which means many bitcasts would become runtime-known if
pointers were involved, or would turn `undefined` values into `0xAA`.
The new logic works by "flattening" the datastructure to be cast into a
sequence of bit-packed atomic values, and then "unflattening" it; using
serialization when necessary, but with special handling for `undefined`
values and for pointers which align in virtual memory. The resulting
code is definitely slower -- more on this later -- but it is correct.
The pointer access and bitcast logic required some helper functions and
types which are not generally useful elsewhere, so I opted to split them
into separate files `Sema/comptime_ptr_access.zig` and
`Sema/bitcast.zig`, with simple re-exports in `Sema.zig` for their small
public APIs.
Whilst working on this branch, I caught various unrelated bugs with
transitive Sema errors, and with the handling of `undefined` values.
These bugs have been fixed, and corresponding behavior test added.
In terms of performance, I do anticipate that this commit will regress
performance somewhat, because the new pointer access and bitcast logic
is necessarily more complex. I have not yet taken performance
measurements, but will do shortly, and post the results in this PR. If
the performance regression is severe, I will do work to to optimize the
new logic before merge.
Resolves: #19452Resolves: #19460
Notably, this improves string printing from
`@as(*[5:0]u8, &@as([5:0]u8, "hello".*))` to `@as(*[5:0]u8, "hello")`,
omitting the pointless ref-deref pair.
Legacy anon decls now have three uses:
* Type owner decls
* Function owner decls
* `@export` and `@extern`
Therefore, there are no longer any cases where we wish to explicitly
omit legacy anon decls from the binary. This means we can remove the
concept of an "alive" vs "dead" `Decl`, which also allows us to remove
the separate `anon_work_queue` in `Compilation`.
This commit also performs some refactors to `TypedValue.print` in
preparation for improved comptime pointer access logic. Once that logic
exists, `TypedValue.print` can use Sema to access pointers for more
helpful printing.
This commit also implements proposal #19435, because the existing logic
there relied on some blatantly incorrect code in `Value.sliceLen`.
Resolves: #19435
Good riddance!
Most of these changes are trivial. There's a fix for a minor bug this
exposed in `Value.readFromPackedMemory`, but aside from that, it's all
just things like changing `intern` calls to `toIntern`.
Perhaps someday, we will make Sema operate on mutable values more
generally. For now, it makes sense to split out this representation,
since it is only used in comptime pointer accesses.
There are some currently unused methods on `MutableValue` which will
be used once I rewrite the comptime pointer access logic to be less
terrible.
The commit following this one will - at long last - delete the legacy
Value representation
`Decl` can no longer store un-interned values, so this field is now
unnecessary. The type can instead be fetched with the new `typeOf`
helper method, which just gets the type of the Decl's `Value`.
This issue was causing debug information to sometimes not function
correctly for some local variables, with debuggers simply reporting that
the variable does not exist. What was happening was that after an AIR
body - and thus debug lexical scope - begins, but before any `dbg_stmt`
within it, the `scope` on `self.wip.debug_location` refers to the parent
scope, but the `scope` field on the `DILocalVariable` metadata passed to
`@llvm.dbg.declare` points, correctly, to the nested scope. I haven't
looked into precisely what happens here, but in short, it would appear
that LLVM Doesn't Like It (tm).
The fix is simple: when we change `self.scope` at the start or end of an
AIR body, also modify the scope on `self.wip.debug_location`. This is
correct as we always want the debug info for an instruction to be
associated with the block it is within, even if the line/column are
slightly outdated for any reason.
This test has regressed due to a bug in the self-hosted COFF linker.
Jakub recommended disabling it for now, since the COFF linker is being
effectively rewritten, so there is little point in fixing the bug now.
This commit changes how we represent comptime-mutable memory
(`comptime var`) in the compiler in order to implement the intended
behavior that references to such memory can only exist at comptime.
It does *not* clean up the representation of mutable values, improve the
representation of comptime-known pointers, or fix the many bugs in the
comptime pointer access code. These will be future enhancements.
Comptime memory lives for the duration of a single Sema, and is not
permitted to escape that one analysis, either by becoming runtime-known
or by becoming comptime-known to other analyses. These restrictions mean
that we can represent comptime allocations not via Decl, but with state
local to Sema - specifically, the new `Sema.comptime_allocs` field. All
comptime-mutable allocations, as well as any comptime-known const allocs
containing references to such memory, live in here. This allows for
relatively fast checking of whether a value references any
comptime-mtuable memory, since we need only traverse values up to
pointers: pointers to Decls can never reference comptime-mutable memory,
and pointers into `Sema.comptime_allocs` always do.
This change exposed some faulty pointer access logic in `Value.zig`.
I've fixed the important cases, but there are some TODOs I've put in
which are definitely possible to hit with sufficiently esoteric code. I
plan to resolve these by auditing all direct accesses to pointers (most
of them ought to use Sema to perform the pointer access!), but for now
this is sufficient for all realistic code and to get tests passing.
This change eliminates `Zcu.tmp_hack_arena`, instead using the Sema
arena for comptime memory mutations, which is possible since comptime
memory is now local to the current Sema.
This change should allow `Decl` to store only an `InternPool.Index`
rather than a full-blown `ty: Type, val: Value`. This commit does not
perform this refactor.
See comment. This slightly regresses a previous fix from this branch. A
proper solution will come soon, with the splitting up of `Decl` into
`Cau` and `Nav`.
This makes tracking easier across incremental updates: `scanDecl` can
now tell whether an existing decl in a namespace was mapped to the one
it's analyzing in the new ZIR.
There is no reason to perform this detection during semantic analysis.
In fact, doing so is problematic, because we wish to utilize detection
of existing decls in a namespace in incremental compilation.
This fixes a bug where, at least with the LLVM backend, `@extern` calls
which had the same name as a normal `extern` in the same Zcu would
result in the `@extern` incorrectly suffixing the identifier `.2`.
Usually, the LLVM backend has a system to change the generated globals
to "collapse" them all together, but it only works if `updateDecl` is
called!
* `linux.IO_Uring` -> `linux.IoUring` to align with naming conventions.
* All functions `io_uring_prep_foo` are now methods `prep_foo` on `io_uring_sqe`, which is in a file of its own.
* `SubmissionQueue` and `CompletionQueue` are namespaced under `IoUring`.
This is a breaking change.
The new file and namespace layouts are more idiomatic, and allow us to
eliminate one more usage of `usingnamespace` from the standard library.
2 remain.
Some of the structs I shuffled around might be better namespaced under
CONTEXT, I'm not sure. However, for now, this approach preserves
backwards compatibility.
Eliminates one more usage of `usingnamespace` from the standard library.
3 remain.
Thanks to Zig's lazy analysis, it's fine for these symbols to be
declared on platform they won't exist on. This is already done in
several places in this file; e.g. `pthread` functions are declared
unconditionally.
Eliminates one more usage of `usingnamespace` from the standard library.
4 remain.
Searching GitHub indicated that the only use of these types in the wild is
support in getty-zig, and testing for that support. This eliminates 3 more uses
of usingnamespace from the standard library, and removes some unnecessarily
complex generic code.
This usage of `usingnamespace` was removed fairly trivially - the
resulting code is, IMO, more clear.
Eliminates one more usage of `usingnamespace` from the standard library.
This is a trivial change - this code did `usingnamespace` into an
otherwise empty namespace, so the outer namespace was just unnecessary.
Eliminates one more usage of `usingnamespace` from the standard library.
Previously, when multiple modules had builtin modules with identical
sources, two distinct `Module`s and `File`s were created pointing at the
same file path. This led to a bug later in the frontend. These modules
are now deduplicated with a simple hashmap on the builtin source.
This fixes an issue with the implementation of #18816. Consider the
following code:
```zig
pub fn Wrap(comptime T: type) type {
return struct {
pub const T1 = T;
inner: struct { x: T1 },
};
}
```
Previously, the type of `inner` was not considered to be "capturing" any
value, as `T1` is a decl. However, since it is declared within a generic
function, this decl reference depends on the context, and thus should be
treated as a capture.
AstGen has been augmented to tunnel references to decls through closure
when the decl was declared in a potentially-generic context (i.e. within
a function).
This implements the accepted proposal #18816. Namespace-owning types
(struct, enum, union, opaque) are no longer unique whenever analysed;
instead, their identity is determined based on their AST node and the
set of values they capture.
Reified types (`@Type`) are deduplicated based on the structure of the
type created. For instance, if two structs are created by the same
reification with identical fields, layout, etc, they will be the same
type.
This commit does not produce a working compiler; the next commit, adding
captures for decl references, is necessary. It felt appropriate to split
this up.
Resolves: #18816
Namespace types (`struct`, `enum`, `union`, `opaque`) do not use
structural equality - equivalence is based on their Decl index (and soon
will change to AST node + captures). However, we previously stored all
other information in the corresponding `InternPool.Key` anyway. For
logical consistency, it makes sense to have the key only be the true key
(that is, the Decl index) and to load all other data through another
function. This introduces those functions, by the name of
`loadStructType` etc. It's a big diff, but most of it is no-brainer
changes.
In future, it might be nice to eliminate a bunch of the loaded state in
favour of accessor functions on the `LoadedXyzType` types (like how we
have `LoadedUnionType.size()`), but that can be explored at a later
date.
mlugg