Meta has unveiled AudioCraft, a new set of AI tools designed to generate high-quality, realistic audio and music from text input. This technology allows professional musicians to explore new compositions without playing any instruments and enables small business owners to effortlessly add soundtracks to their videos or advertisements.
AudioCraft comprises three models: MusicGen for creating music, AudioGen for generating sound effects, and EnCodec as a generative AI decoder. MusicGen was trained on approximately 400,000 recordings along with text descriptions and metadata, amounting to 20,000 hours of music owned by Meta or specifically licensed for this purpose. The model’s complexity allows it to generate coherent samples for long-term musical structures, making it ideal for producing novel musical pieces. Meta envisions MusicGen evolving into a new type of instrument, similar to the emergence of synthesizers.
AudioGen, on the other hand, was trained on public sound effects and can produce various environmental sounds, such as a dog barking, cars honking, or footsteps on a wooden floor. Additionally, Meta released an improved version of the EnCodec decoder, enabling higher-quality music generation with fewer artifacts.
The company is making the AudioCraft models available as open-source code to facilitate access for researchers and practitioners. They hope that by sharing the code, others can train their own models using diverse datasets, thus advancing the field of AI-generated audio and music. However, Meta acknowledges that the current datasets used in AudioCraft lack diversity, primarily featuring Western-style music, and the text and metadata are written in English. To address this limitation, they encourage researchers to explore new approaches to limit potential bias and misuse of generative models.