You can turn off Voice and Audio Activity on your Google account at this link, although this won’t stop Google from recording and archiving anything you ask Google Now. Instead, your voice recordings will be made anonymous without any direct links to your Google account.
Source: Google Now tips and tricks: Audio History records everything you say | BGR