Here's how wake words work: the speaker is always recording its surrounding sounds, and when it hears its wake word or phrase, it sends a recording of the audio that follows to be processed by the servers of its parent company, and formulates a response. If the user doesn't use the wake word, there shouldn't be any transmission of data – so theoretically, Amazon, Google, et al. are not logging and analyzing every single conversation a user has in front of the speaker.
However, this is not a perfect system. A bug was discovered in March of 2018 with Amazon's Alexa being accidentally triggered to laugh – and creeping out its users when a disembodied chuckle was emitted from their devices unprompted.