well - probably first step is to show error. You need a stack of VRAM to run this. I didn't bother with it for this reason. > 12gb of VRAM. cards that are suitable are k80 - 24gb/32gb of VRAM / RTX 5000 - 16gb RTX 6000 / RTX 8000. Not even the 2080ti is appropriate at 11gb. the just released nvidida 3090 - the latest / 24gb of VRAM. AMD cards will not work. Wait another month - get the 3090 card or two / there's a limit of these cards being released. This card is better than the one below. (don't get the 3080 it can't be joined together with nvlink) - you'll be in a better position to get anywhere. You'll need a 750Watt power supply. I recommend a HP workstation.

"On a V100, it takes about 3 hrs to fully sample 20 seconds of music"
US $8,399.00
https://www.ebay.com.au/itm/nVIDIA-Tesla-V100-GPU-Accelerator-Card-16GB-PCI-e-Machine-Learning-AI-HPC-Volta-/153223723116

(Alternative is to run it with google colab - but this is problematic.)

@perlman-izzy why do you want it so bad?

@perlman-izzy What's your email address? Sounds like you are using my docker image so I am happy to keep helping you. I also completely understand the desire to do so - jukebox has been the most fun with music I've had in a very long time and the results I'm getting in my own domain (electronic music) are incredible 😭 I've spent pretty much every day since jukebox released editing the code to support various things that I need for better co-composition including the ability to finetune to the 5b model, much larger batch sizes for sampling, editing of tensors for composing before they are rendered as audio, etc. I think I can get you to the same place pretty easily without asking for any money, so let me know and we can get you rolling!

@johndpope I have access to a GPU with 24gb 'per-GPU' ram (which I think is the same as VRAM?). With that amount of power, how long do you reckon it would take to process 6 min? 10 min?

johndpope last error message was : file "jukebox/sample.py", line 279, in fire.Fire(run) File
"/usr/local/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name) File
"/usr/local/lib/python3.7/site-packages/fire/core.py

@btrude Any chance you can make those changes available? I'd like to roll that stuff into the google Colab I've been using.

@bsturk Possibly... I'm not at all familiar with the colab setup as I've been running it locally since may. If the colab simply clones the jukebox repo then it will be a bit difficult as all the changes I'm describing are at the sample.py and train.py levels - though we can definitely figure something out if you can link me to the file you're using. Finetuning the 5b requires 48484mb of VRAM without using the distributed model (ie its only possible to do it with one RTX 8000 at the speed and batch size [1] that such a setup implies) so I don't think you will be able to do that much with colab, but maybe they offer RTX 8ks? Either way, finetuning the 1b_lyrics model is possible with 1/3rd the VRAM of the 5b and yields awesome results as well so not too much of a loss and hopefully I didn't get your hopes up mentioning the 5b. The co-compositional changes I've made are actually just realizing features that are described in the paper but weren't actually shipped with this repo's code, so I have to wonder if some of those features are present in the colab perhaps? Specifically though, I've made most input parameters hyperparameters safely accessible in the code via calls to hps.get('some_param') (so you can specify genre/artist/temperature from the command line and different prompts per-level), and then implemented a new mode truncate and an additional parameter pref_codes which will duplicate the specified codes for subsequent continuations/upsampling. All of this allows you to do as the paper describes, creating an initial batch of n_samples at 1x n_ctx, then listening to the samples and choosing your favorites, then setting them as the pref_codes values and creating multiple different continuations of the same codes, and so on (with the added bonus of being able to shave off unwanted parts of the samples using truncate). All of that is to say let me know if these are actual things you need and I will gladly share the code/modify your colab/whatever.

I have access to a GPU with 24gb 'per-GPU' ram (which I think is the same as VRAM?). With that amount of power, how long do you reckon it would take to process 6 min? 10 min?

@perlman-izzy I sent you an email about your other issue, but an rtx titan doing levels 2, 1, and 0 will take ~48 hours using the default hop fractions for 6 minutes of music. It's hard to say how long it takes to do each level individually, but its probably somewhere around 4-6 hours for level 2, 8-12 hours for level 1, and 30 or so for level 0. All of this depends heavily on n_samples and whether or not you are doing any sort of editing/pruning of codes.

@perlman-izzy - you can type nvidia-smi to show card. (There are spot prices on ec2 - $.027 for k80 gpu - 24gb - you can get on aws which are cheaper - but they can terminate unexpectedly but you pay 10% of regular price. Although they have heaps of V(ideo)Ram - the k80s that are provisioned on lower end of aws are really slow https://technical.city/en/video/GeForce-RTX-2080-Ti-vs-Tesla-K80 - the current card 2080ti is 333233.3% faster. Again - the latest 3090 cards have 10,000 cuda cores / and if you were to get 2 - you'd be 48gb / and could handle fine tuning larger model. a tid bit - I think the line
"On a V100, it takes about 3 hrs to fully sample 20 seconds of music" - doesn't just reflect a single file of 20 seconds - but a whole suite. happy to be corrected here.
also - @perlman-izzy - presumably - you are using ubuntu? docker + windows + nvidia (historically) don't play well together. you'd be better off with a dedicated ssd drive to boot to with ubuntu 18 / (don't go for ubuntu20 just yet).

you'd be 48gb / and could handle fine tuning larger model

@johndpope This is unfortunately not true. Jukebox as released here uses distributed data parallel which requires the model be duplicated across each GPU. I've experimented with this extensively and as far as I can tell the only way to finetune the 5b (5b_lyrics is too large) is with an RTX 8000 removing DDP or, as mentioned in the readme, with some custom implementation of gpipe.

@btrude - I stand corrected. So, to train on aws /ec2 with (SPOT PRICING / can terminiate at any instant) p3.8xlarge (64GB of VRAM) is $3-4 /hr in N. Virginia ~ $100 USD/ day
https://aws.amazon.com/blogs/aws/new-amazon-ec2-instances-with-up-to-8-nvidia-tesla-v100-gpus-p3/
That gets you 8x v100 cards. The lower model down has less ram - p3.2xlarge | 1 | 16 GiB won't handle fine tuning larger model. @btrude - did you upload any of your creations to soundcloud?

We successfully fine-tuned the 1b model in the cloud on p3.8xlarge, on a custom dataset, over a few days.

@btrude - I stand corrected. So, to train on aws /ec2 with (SPOT PRICING / can terminiate at any instant) p3.8xlarge (64GB of VRAM) is $3-4 /hr in N. Virginia ~ $100 USD/ day
https://aws.amazon.com/blogs/aws/new-amazon-ec2-instances-with-up-to-8-nvidia-tesla-v100-gpus-p3/
That gets you 8x v100 cards. The lower model down has less ram - p3.2xlarge | 1 | 16 GiB won't handle fine tuning larger model. @btrude - did you upload any of your creations to soundcloud?

Not yet, no. Hoping to do so soon.

@btrude I'm using the colab located here

#72

and yeah, I'm not sure I could utilize the 5b changes you've made in Colab, I'd be interested in the 1b changes though.

@bsturk In trying to implement this I'm finding that the colab linked here is just too janky and at odds with sample.py to just do a clean copy and paste so I'm going to have to hold off for now. It's really weird to me that the colab duplicates code in sample.py when its possible to make calls from the command line from within colab. I also don't think the colab as is really elucidates how the larger sampling process works either, so its not like there is any benefit at all to using it like this. I honestly feel bad for everyone using it because it would be 100x simpler if you replaced all the python with a few example shell commands like those in the readme and then made the relevant parameters into fields on hps. That would take some work, but I have a small demo working in colab already. Now it would just be a matter of providing my sample.py as a gist and then using it instead of openai's. Let me know if you would like to test this and I'll go ahead and put it together.

@bsturk I went ahead and just made it, you can load it through github in colab with the link below

https://gist.github.com/btrude/72c788b8ce8c8b2f8bc8ecf309a04f24

this is only for sampling, but has all of the features I mentioned with notes on usage. exiting on an explicit level shows an error message but the codes/audio are still there and can be downloaded from the filebrowser on the left.

@bsturk --sample_length_in_seconds = the length of the sample you're generating/continuing/truncating/upsampling (you have to increment this as you continue sampling, etc)
--total_sample_length_in_seconds = total intended length of the song, though you don't actually have to sample up to that point

I'm assuming after doing the above with levels = 1, once I have what I want, I can bump that up to 2 and it will utilize the previous iterations saved on gdrive?

Right, you just need to replace the --codes_file=/whatever/path with the path to the new codes file you made with --mode=upsample. You can find it in the filebrowser and right click it to get the path then just copy and paste it into the colab cell.

thanks again for making this available...

My pleasure. I like this setup a lot now that I can easily match my local setup in colab. I think once google replaces their non-colab V100s with A100s the P100s of colab will probably be replaced with V100s which will make this even better.

Hi, I want to know how long it need to generate a song ?
3 hours for 20s ? Is that true ?

Hi, I want to know how long it need to generate a song ?
3 hours for 20s ? Is that true ?

For 1b_Lyrics model is around 1 hour to generate 20s song on 2080ti. And It seems 2080ti is not good enough to run 5b model

@btrude Sorry, this is a noob question, but is it already possible to generate samples with your modified notebook or is it only theoretical at this point? I seem to get stuck when I run your "correct example" cell. What do I have to do after that to generate samples?

@btrude Sorry, this is a noob question, but is it already possible to generate samples with your modified notebook or is it only theoretical at this point? I seem to get stuck when I run your "correct example" cell. What do I have to do after that to generate samples?

It is possible and I just tested the cell you described and it works. Run the first 3 cells (the second one requires authentication!), then the one you mentioned. Keep in mind that the models have to be downloaded the first time you run jukebox and that takes a few minutes. If you keep having issues please provide logs or screenshots.

@btrude Just tried again, 1b works fine but 5b just kinda stops (runs out of memory?). Sorry if this has been established before in this thread, it's all a bit confusing for me.

@btrude Just tried again, 1b works fine but 5b just kinda stops (runs out of memory?). Sorry if this has been established before in this thread, it's all a bit confusing for me.

All of the notebooks going back to the original by openai have this issue unfortunately. Either you can try to luck into an instance with more memory or you can pay for their premium service.

Got it, thanks for the quick responses!

Hi @btrude I have a few questions about how to generate VQ-VAE codes without the priors. You can email me at username at gmail.com

Hi @btrude I have a few questions about how to generate VQ-VAE codes without the priors.

This is not something I have direct experience with, though I intend to implement a --mode=vqvae at some point soon to support a pure upsampler (which I'm guessing is your goal here as well). As you may already know the vqvae creates codes for input tracks and then passes them to the priors during training so I've always considered this to most likely be a trivially easy task - I just haven't had time to implement it yet.

Final solution: Use Google colab and do not train on your own custom dataset.
Yes. Training on your own dataset is not required if you just want to get a quick & dirty continuation out. This does only work for continuations.

You just need to set your genre to the genre of the song you choose and set the singer or artist to unknown. it'll work.

If you want to create songs you probably need way more songs then you'd think. If you only use a few, the algorithm is probably going to overfit.

@btrude What's your email address,I have a question for you, my email [email protected]