llb.State.Marshal uses a vertexCache map (keyed by the llb.Vertex interface
value) to memoize which vertexes it has already visited while marshalling.
However, before this change, llb.DefinitionOp was constructing new pointers for
each value in the return slice of Inputs(), which meant that each input had a
different key in the vertexCache (due to being different pointer values). This
meant no memoization actually occured. I noticed this while using
llb.DefinitionOp with a fairly large graph when my program crashed after using
>16GB of RSS during a call to llb.State.Marshal.
The fix here avoids changing vertexCache as that would impact many other
vertex implementations. Instead, just DefinitionOp.Inputs() is updated to cache
the pointers it creates and pass that cache to its descendents. This fix
resulted in the program that was previously crashing out-of-memory to run
without any perceivable increase in RSS.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Erik Sipsma
Tonis Tiigi
Edgar Lee