Hey MarkPierce,
There are some limitations, but ultimately, the waters are shared. As a boater, I often like to go the shortest distance from Point-A to Point-B allowed by my draft. That isn't always through a marked channel. Once I get out of a marked channel, I could well be in good fishing ground. Once that happens, it is my job to keep my boat from destroying someone else's trap and buoy, not the other way around.
It is also the case that, sometimes, traps drag or buoys break lose, e.g. in storms. In those cases, perhaps despite the best intention of their owners, they end up where they shouldn't. I suspect most such cases result from owner's not promptly collecting their traps and inspecting their gear, but I don't know. In some cases the buoys are research buoys that are left out for a long time to collect measurements, etc.
There was also one time I was running in the ICW in Florida between St. Pete and Clearwater. I was enjoying a nice leisurely ride in the channel as I thought I suddenly saw a pop-up radar contact coming fast right as I turned a bend. I barely got of 5-taps of my horn before I saw a Hatteras sport fisher coming around, very, very fast, right at me. I was out of options and cut the throttles and drifted to the edge of the channel to let it through. My aft drifted out of the channel.
In doing so, I apparently picked up a crab trap, or a piece of rope from one, right outside of the channel. I finished the trip going slowly, with that engine at idle speed and rudder trim, mostly using that engine just for docking. It was a thin little rope and I suffered no damage at all, thankfully. I got lucky.
I was really annoyed at the fisherman for putting traps right up to the channel's edge like that. Of course, that was actually the fisherman's right and many folks did. And, of course, it wasn't my right to drive over one of them. The fisherman was the only person who lost anything that day. The buoy's tag wasn't on the rope I picked up, so I couldn't make that right for the person, either.
Ultimately, the Hatteras was the one in the wrong in the situation. I suspect they were just trying to get out to the gulf fast to go fishing and didn't want the lazy ICW to cost them a good chunk of their day. None-the-less, they had no right or privilege to drive like that and were a danger.