Using LightweightPipeline.SetRenderTarget unilaterally considers intermediate targets as being texture arrays, which is not true for the shadow maps.
The seperate depth RT wasn't being created or managed correctly in stereo.