toma and cade: two SAPHOs passing a value¶
Superseded in part (2026-10-11): a processor now has several links. The source forms are
toma(k, x),cade(k),fcade(k)and the_syncforms, the pins carry the link number (toma_k,cade_dado_k, …), andLINKASis gone: every link is synchronized unless the program uses a_syncform. Seetoma-and-cade-links.md. The protocol, the hardware per link and the checks below still hold. The appcomp named below is asmcomp’s first pass since v7.1 (2026-10-13).
Design record of TODO item 20 (2026-10-07, target v7.0). Worked out with Luciano in conversation; nothing below is implemented yet. Sections 1-4 are the design as agreed, section 5 what was checked in the code and how, section 6 what is still open, section 7 the alternatives that were weighed and why they lost.
1. What the user writes¶
One SAPHO hands a value to another; each side waits for the other, nothing is lost and nothing is read twice:
// writer // reader
toma(x); y = cade();
toma(x) does not return until the partner has taken x; cade() does not
return until the partner has offered a value. Whoever arrives first waits
(a synchronous channel, the rendezvous of Go’s unbuffered channels and of
Occam). There is no port number and no status port: each processor has one
such link, to one partner, usable in both directions. The ordinary I/O ports
(in, out, req_in, out_en) are untouched and never wait.
C+- and C++ both get toma(x), cade() (the word as an int) and fcade()
(the word as a float), as in()/fin(). As the whole right-hand side of an
assignment (y = cade();) the word goes into y as it is, with y’s type;
inside an expression, an index or an argument (z + cade(), out(0, fcade())) it is an int or a float operand. CAD has no push form: when the
accumulator is in use, the compilers write PSH before it (one word, one
cycle; the waiting loop jumps back to CAD, not to the PSH). Decided
with Luciano 2026-10-08, rather than a P_CAD opcode, which would need
another renumbering below 32.
2. The protocol: one bit per side¶
Each side owns one bit that only it flips, and reads the partner’s:
the writer owns T (“toma”): it flips T when it offers a value;
the reader owns C (“cade”): it flips C when it takes the value;
a value is waiting while T != C; it has been taken when they are equal again.
The writer also needs a flag W (“waiting”) so that T flips once per
toma, not on every turn of its waiting loop.
writer, |
reader, |
|
|---|---|---|
goes on when |
W is set and the partner’s C == T |
the partner’s T != C |
otherwise |
if W is clear: flip T, set W; jump to itself |
jump to itself |
when it goes on |
clear W |
take the input, flip C |
The two orders:
tomafirst. The writer flips T and loops. The reader’sCADsees T != C, takes the value, flips C and goes on. The writer’s next turn sees C == T and goes on.cadefirst. The reader sees T == C and loops. The writer arrives, flips T and loops. The reader’s next turn sees T != C, takes the value, flips C and goes on. The writer then sees C == T and goes on.
In both orders the writer leaves only after the value was taken, so no buffer is needed: the value stays on the writer’s output bus while it loops (section 5). The bits are levels held in flip-flops, never one-cycle pulses: two processors spinning in loops see each other’s instruction only one cycle in every turn, and with equal loop lengths out of phase they would never meet.
Reset clears T, C and W on both sides.
3. Hardware¶
Two instructions, TOM and CAD, each a conditional jump to its own
address, decided where JMP/JIZ are decided (the prefetch, section 5).
The operand is the instruction’s own address, written as a label, as in
@fim JMP fim (@L TOM L); neither needs a data operand. Neither has a
row in instr_dec.v (like JMP/JIZ): an opcode without one decodes to
“pass the accumulator through the ALU”, which is what a looping TOM needs.
CAD loads the accumulator through INN’s path (ula_in2_ctrl) with
cade_dado in place of io_in, raising no req_in outside.
Opcodes (agreed 2026-10-07). Flow control is kept below opcode 32
(check_isa.py), and 0-31 are all taken: TOM = 20 and CAD = 21, right
after RET, every opcode from ADD up moving by 2 (the highest becomes
107, still 7 bits). An encoding break, in line with v7.0.
Generated only when used (pay for what you use): a program with no
TOM gets no T, no W and no TOM pins; with no CAD, no C and no CAD
pins.
Pins of the generated processor:
pin |
direction |
width |
with |
what |
|---|---|---|---|---|
|
out |
1 |
|
my T |
|
in |
1 |
|
the partner’s C (has it been taken?) |
|
out |
1 |
|
my C |
|
in |
1 |
|
the partner’s T (is a value offered?) |
|
in |
NUBITS |
|
the partner’s offered value |
|
out |
NUBITS |
|
my offered value ( |
Wiring two processors (A sends to B; the other direction is the same with A and B swapped), wires only, no user logic:
A.toma -> B.taqui B.cade -> A.valeu A.toma_dado -> B.cade_dado
Both processors must have the same word format: the same #NUBITS
for ints, and the same #NBMANT/#NBEXPO for floats. The link carries the
word as it is, with no conversion: with different widths Verilog joins the
buses by cutting or zero-filling, so a negative int arrives wrong (deduced
from Verilog’s rules, not tested), and a float of another format arrives
garbled. Nothing warns: each program is compiled alone (nipscernlab/aurora#240).
On one clock A.out -> B.cade_dado is the same word (the wiring used
before toma_dado existed). On two clocks see section 8: toma_dado and
#(.LINKAS(1)) on both.
cade_dado is a pin of its own: inside the reader a mux picks in or
cade_dado by the instruction (INN or CAD), hidden from the outside.
Sharing the in pin would need a mux in the user’s top, which cannot know
when to switch (CAD raises no req_in).
The only signals crossing between the processors are T and C, each straight out of a flip-flop and into the other’s jump decision: no combinational path runs from one processor into the other. No Fmax loss is expected from the link (deduced from the structure; to be measured, section 6).
4. Compilers and assembler¶
toma(e)-> the value ofein the accumulator, then@L TOM L.y = cade();->@L CAD L, thenSET y.asmcomp and appcomp: the two mnemonics (operand class
code, the own label);isa.tsv,asm_share.c,core.v’s flow localparams andcheck_isa.pyin step. appcomp’s lexer (app.l) is a sixth copy of the mnemonic list: it counts the instructions that size the instruction memory, and withoutTOM/CADthereMINSTScame out one short per use (the last instruction fell off the memory).check_isa.pynow holdsapp.ltoASMComp.ltoo.hdl.c: the pins above in the generated top, only with the instruction present. The standalone testbench plays the partner, so each side can be simulated alone, as
in()/out()already are withinput_N.txt/output_N.txt:cade()takes the words ofSimulation/cade.txtone by one (when the file ends it says so and the nextcade()waits forever), and everytoma()value is written toSimulation/toma.txt. It keeps the same protocol (it owns T forcade, C fortoma).asmcomp checks that a
TOM/CADoperand is a label of its own line (@L TOM L); any other target is rejected (“waits by jumping to itself”).cmmcomp (diff shown to Luciano first):
tomaandcadeare reserved words, andfcade.toma(e);is a statement (STMT_TOMA,exec_toma);cade()/fcade()are stdlib expressions (OP_STD_CADE/OP_STD_FCADE). As the whole right-hand side of an assignmentSTMT_ASSIGNtakes them (exec_cade: the word lands in the variable with its type); anywhere elseexec_cade_exprgives an int / float operand, withPSHfirst when acc is live (the same choice asINN/P_INN). A comp destination or a comptomais rejected (one word). No new grammar conflict.cppcomp: the same builtins. The scalar store and the local initializers set a flag that
gen_expr_toconsumes at once: therex = cade();puts the word in as it is; anywhere elsecade()/fcade()is an ordinary int/float call (@L CAD L), and the expression code pushes a live acc by itself.tomaof a wider value is rejected.C++ inline assembly may write
TOM/CADdirectly.Tests:
Compilers/CMMComp/Tests/link_*(the four programs) and theLINKproject pass inScripts/regress.sh(two pairs of processors, checked against the sim golden and against the sequences n7+3 and n+1 computed by the script);NegTests/cade_compfor a comp destination.link_pongwritestoma(um * um + cade())(thePSHpath) andlink_cpp_pingout(0, cade());fcade()was checked by hand with real floats across the compilers (C+- hands n1.5, C++ takes0.5f + fcade(), 16 of 16 right).TOMandCADare insapho_all’sblocks_except.txt(alone they would wait forever). The same pass also runs both scenarios across the compilers: the C+-link_enviahands to the C++link_cpp_recebe, the C++link_cpp_pingtalks to the C+-link_pong(sources inTests/LINK/cpp/, built inside the pass with the CPP phase’s commands). The C++ rejected forms were checked by hand.
5. Checked in the code, then in simulation (2026-10-07)¶
Read in SAPHO/core.v first, then confirmed by three simulations of two
processors from hand-written assembly (Icarus, outside the regress):
test |
result |
|---|---|
writer offers |
16 values, in order, none wrong; 1709 cycles, |
no waiting loops on either side |
16 values right, 260 cycles (~16 a value) |
one link both ways: ping offers i and takes back i+1, pong takes, adds 1, offers back |
16 answers right |
What the reading said:
The writer’s output bus holds the value while it loops.
outis the ALU output (mem_data_wr = ula_out), not the accumulator, which is why it changes on every instruction. Instructions that leave the accumulator alone (OUT,JMP,NOP) pass it through the ALU, so a repeatingTOMdecoded the same way keepsxon the bus.When things happen. A jump is decided in the cycle the instruction leaves instruction memory (
prefetch); in that cycle the ALU executes the instruction before it, which is whyJIZtestsula_outand not the accumulator (if_acc = |ula_out). ForLOD x; TOM: in the cycleTOMis decoded the ALU executesLOD x, soxis already on the bus; T flips at the end of that cycle andxstays.CADsamples its input at the end of the cycle it goes on (ior <= io_in, as forINN), the cycle C flips.Nothing after a looping instruction runs. The next fetch address is computed in the same cycle as the decision; while
TOM/CADloops the address fetched is its own, as with@fim JMP fim.
6. Open¶
The interrupt: decided, stays as it is (Luciano, 2026-10-11). The interrupt restarts at
ITRADDwithout clearing T, C or W; a restart in the middle of atomaleaves T flipped with nobody holding the value and the pair out of step. Not interrupting a processor while it waits intoma/cadeis the program’s responsibility, the same line as the deadlock below. Two alternatives were set aside for v7.0 and wait for a real case: holditruntil the wait ends (small hardware, but an absent partner then blocks the interrupt forever), or reset the link on both sides (a wire between the partners).Deadlock: both sides in
toma(or both incade) wait forever, as in Go; the program’s responsibility. A simulation could flag it.Two clocks: implemented in simulation (
LINKAS, section 8); the board check and the Quartus constraints are open.Fmax: one fit of a two-processor design on each board.
Test plan: two SAPHOs in Icarus; the writer sends 0, 1, 2, … N; the reader checks the sequence. Writer faster, reader faster, alternating speeds (waiting loops of different lengths); both directions on one link. Passes if every value arrives once and in order. Then a fixture in the regress (a project pass, as DTW) and a small example for the boards.
7. Weighed and dropped¶
A FIFO between the two with a status port read by
in()(the first proposal): works without touching the core, but every transfer needs a waiting loop in the program and two ports per side. Dropped for transparency.SAPHO/myFIFO.vwould not have served as is: it writesdataone cycle afterwrreq, when a SAPHO’soutbus has already changed (by reading at the time; confirmed in simulation on 2026-10-09, with two more defects, all fixed: docs/design/myfifo.md).Freezing the core while waiting (a
waitpin): as transparent, but it is the pause logic dropped from#PIPELN(docs/design/pipeln-and-division.md section 2) and puts a combinational path from one processor’s decoder into the other’s stall.One-cycle pulses instead of the two bits: two looping processors can miss each other forever (section 2).
8. Two clocks (LINKAS, implemented 2026-10-09, replaced 2026-10-11)¶
(History: LINKAS was replaced by synchronized links by default and the
_sync forms, toma-and-cade-links.md section 7. The synchronizer and the
toma_dado register described here are the ones every link now has.)
Asked by Luciano on 2026-10-09: can the link join two processors that run on different clocks? Yes, and the protocol (section 2) does not change. Proposed and approved the same day (“segue com suas sugestoes”): the data register (8.2, a) and a module parameter (8.3, option 1). What is a fact of the code, what was measured and what is deduced is marked.
8.1 Why the protocol already fits¶
Fact (code): T and C are levels in flip-flops. Each flips once per value and then holds; there is no one-cycle pulse to miss. This is the two-phase (“toggle”) handshake used to cross clock domains.
Fact (code): the writer leaves
TOMonly after it sees C == T, so its value stays offered for the whole exchange. No buffer is needed.Deduced: across clocks a bit can only arrive late, never be lost. A late T makes the reader wait longer; a late C makes the writer wait longer. Order and “once and only once” hold as in one clock.
8.2 What changed in the hardware (SAPHO/core.v)¶
A two flip-flop synchronizer on each incoming bit. With
LINKAS = 1,taquienters the reader through two flip-flops on the reader’s clock beforecad_goand C use it;valeuenters the writer through two flip-flops on the writer’s clock beforetom_goand W use it.The writer’s value in a register,
toma_dado. Without it the partner would read theoutbus, which is the ALU output (ula_out), combinational (section 5, item 1). Deduced, not measured: in the cycle T flips the ALU goes from the instruction beforeTOMtoTOMitself, and the opcode change may glitch the bus for a few nanoseconds after the edge. WithLINKAS = 1a NUBITS-bit register is loaded withula_outin the cycle T flips (tom_offer, the same edge), so the word the reader takes is a flip-flop output that was already still when T flipped. Dropped alternative: keepoutand ask the user for a timing constraint on it.toma_dadois a pin of every processor that hasTOM. WithLINKAS = 0it is a wire equal toout, so a synchronous link may be wired either way; the older wiringA.out -> B.cade_dadostays valid.Generated only when used.
LINKAS = 0keeps today’s hardware. Yosys onlink_ping(TOMandCAD),synth -flatten, 2026-10-09:previous HDL: 1127 cells;
new HDL,
LINKAS = 0: 1127 cells, identical type by type;new HDL,
LINKAS = 1: 1163 cells, of which +32$_DFFE_PP_(thetoma_dadoregister) and +4$_DFF_P_(the two synchronizers).
8.3 How a link is declared asynchronous¶
Which clocks exist is known in the user’s top-level, not in the program, so
the choice is made there. The generated module of a program with TOM or
CAD (hdl.c) now opens with #(parameter LINKAS = 0) and passes it to
processor and core. Nothing changes in C+-, C++, cmmcomp, cppcomp or
appcomp. A program without TOM/CAD gets no parameter.
Rule: set LINKAS = 1 on BOTH processors of a link whose clocks differ
(the reader synchronizes T, the writer synchronizes C and holds the data),
and wire the data from toma_dado:
link_envia #(.LINKAS(1)) A (.clk(clk_a), .rst(rst_a), .toma(a_toma),
.valeu(b_cade), .toma_dado(a_dado), ...);
link_recebe #(.LINKAS(1)) B (.clk(clk_b), .rst(rst_b), .cade(b_cade),
.taqui(a_toma), .cade_dado(a_dado), ...);
Reset: both processors leave reset together, each with the release
synchronized to its own clock (two flip-flops per domain from one reset
source, as link2_tb.v does). A reset of one side alone leaves T and C out
of step, the same open problem as the interrupt (section 6).
8.4 Checks¶
Simulation (Icarus), in the regress:
Compilers/CMMComp/Tests/LINK/ TopLevel/link2_tb.v, run inside theLINKpass asLINK2. The same programs on clocks of 20 ns and 27.3 ns,LINKAS = 1, three pairs: envia fast -> recebe slow, envia slow -> recebe fast, ping slow <-> pong fast. Checked against the sequences regress computes (n*7+3, n+1). Measured 2026-10-09: 16 of 16 in order on each of the three.What the simulation does not prove. The same testbench with
LINKAS = 0also passes (measured): Icarus has no metastability and no glitches, so it shows the synchronizers do not break the protocol, not that the crossing is safe. That part rests on the synchronizer and on the board.Unchanged:
link_tb.v(one clock,LINKAS = 0) gives the same four outputs as its golden; the standalone testbench of a processor withTOMconnectstoma_dadoand still writes the righttoma.txt.
8.5 Still open¶
Board: a two-clock design on the DE10-Nano / ZYBO, with Fmax: the
LINKALLpair (8.6) is the design to take there. The Quartus constraints to write and check there: a false path (or aset_max_delay) into the first synchronizer stage oftaquiandvaleu(*tom_link.sync.s1,*cad_link.sync.s1), and aset_max_delayof one reader period ontoma_dado -> cade_dado. The exact.sdclines are not written yet.Latency: each value pays about 2 synchronizer cycles on each side on top of today’s ~16 cycles a value (estimated, not measured).
8.6 The whole processor over the link (LINKALL, 2026-10-11)¶
The LINK programs are small; LINKALL puts the whole processor on each
side, the processor used for timing (sapho_all, every block), so the
Fmax and the board check of a linked design measure two full SAPHOs.
Tests/sapho_all_aissapho_allplustoma(t & 15);after itsout();Tests/sapho_all_bissapho_allplusk = cade();before itsin(0)andif (k != x) s = s + 4096;. B’s input port carries A’s last output (a register loaded onout_en, as a board holds it for the LEDs) andcadetakes the same word: the check movess, and so the LEDs, only if the link delivers a different word. Every added line ends in// LINK; regress (LINKALL-source, also under--no-sim) fails if either file differs fromsapho_all.cmmin any other line, so a change tosapho_allmust be copied to both.Tests/LINKALL/TopLevel/linkall_tb.v, three circuits in one run: A -> B on one clock (20 ns,LINKAS0); A (20 ns) -> B (27.3 ns),LINKAS1 on both, the reset released in each domain through two flip-flops; andsapho_allalone with its input = its last output XOR 7 (starts at 7; the XOR keeps it off the fixed point 13 that a plain feedback from 0 falls into).Checked against
model.pyonly, never a blessed golden: A’s lines are model(7) = 4, B’s model(4) = 10, and each fed-back line is model of the previous one XOR 7, the cycle 4, 11, 13, 8, 12, 0. Measured 2026-10-11: pair 1 A 17 / B 16 lines, pair 2 A 13 / B 12, fed back 17, all equal to the model.A chain (A -> B -> C) is not possible: each processor has one link (section 1). A link carries values both ways between its two processors (
link_ping/link_pong), so a ring of two is possible; more than two needs several links per processor (TODO item 23, v7.1).What
sapho_all_a/_badd tosapho_all:TOMon A,CADon B (andLINKASwhen the clocks differ); 936 and 941 instructions against 933.
8.7 For the documentation (issue #40)¶
Superseded (2026-10-11): the user-facing text for v7.1 is the list in
toma-and-cade-links.md section 11 (several
links, no LINKAS). From this note it keeps only:
both processors must have the same
#NUBITS, and the same#NBMANT/#NBEXPOwhen floats are passed: the word goes as it is, no conversion, no warning (section 3);the reset rule when the clocks differ (8.3);
(any clock) an interrupt during a
toma/cadewait puts the pair out of step: do not interrupt a processor while it waits (section 6).