This is exactly why I have ceased with the PFD under elastic cords on the kayak bow to an inflatable right where it belongs. I kayak alone.
I read some statics that state (1) 9 out of 10 drownings occur with the victim is not wearing a PFD per CDC, and (2) drowning is the second leading cause of death for kids under 14 in NH, per NHwatersafety, and the one that really grabbed my attention, (3) 90% of drownings die within 30 feet of safety, per NHwatersafety.
I used to feel that no matter what, I can always swim a bit to get to help or safety. I no longer think this way. Despite what you see on TV, the victim almost never cries for help or waves their arms in distress. Help could be a few feet away and not know you are drowning.
http://www.nhwatersafety.com/drowningprevention.htm
Safe boating!