My first approach was to go buy a proper photodiode, then I thought, well I don't really need to compare to anyone, I just need this to work. So I thought about scratching the phosphor off. But when I realized just how hard that is, I just gave it a go without modifying the darn thing at all.
Just take a blue power LED with a decently-sized die. That's basically the same as your white LED with the phosphor removed (again, it's not a UV LED hiding underneath the phosphor).
But then I realized that any light from outside that makes the phosphor glow, will then emit some light of a different kind inwards to the diode.
Which for your measurement is basically irrelevant. The net efficiency is probably significantly higher with the phosphor removed. Keep in mind that a LED used as a photodiode is sensitive to wavelengths shorter than its peak wavelength when used as an emitter. So the blue diode inside a white LED is sensitive to UV, and in your use case, all the phosphor does is convert a little UV to green and red light. This is essentially lost to the sensor diode, which is blind to those longer wavelengths.
So in short, you're better off with a simple blue LED. Of course, white LEDs are plentiful, so there's a practical reason why you might prefer those; that's fair enough.
Or just build a simple transimpedance amplifier. Plenty of circuits all over the net.and you can add more diodes in series to get higher voltages that are less affected by the impedance
there is modulation...not unexacted
Not sure what you mean here. Many light sources will indeed flicker at e.g. 50/60Hz grid frequency, or at a substantially higher frequency depending on how they're driven.