Fragmentation becomes an issue when you're either building services that run in the background or interact with hardware (BT, WiFi, WiFi-direct, Camera, GNSS, Gyro/Accel). I feel fragmentation hard with my app.
For hardware, I believe that 100%. I am talking about "regular" android apps that don't need deep access to the hardware. By deep I mean something harder than recording something (e.g. taking/picking a picture is delegated to the default system app, not the one you build) or displaying the current position with a mapping sdk.
Bluetooth in particular I can believe. It is amazingly complex and adding the OEMs customizations on top must be a total pain.