This comment gave me the best hint so far to solve the SSR issue without actually doing SSR.
I didn’t implement this in my own app yet, but I will soon; unless I misunderstand him, the idea is to have prerendering (locally or via prerender.io) in place, and to use the page snapshots in conjunction with the SSR API onPageLoad(). In other words, this is SSR without an actual server side rendering. This should solve your FCP and LCP problem at the same time. Let me know what you think.