When you go out in a boat on the ocean, isn’t there a worry about whales and such?

I just moved to the ocean and I don't understand it. There is this whole culture here... and people go out on these little fishing boats on the ocean, and sometimes the whales are in the bay! Help me understand this or do they know the whale's seasonal cycles and go at different times?