Video memory usage

Hello there,

I’m actually writing a scenario for an interactive movie and trying to play the video and sound from score. I split each sequence into separate video and audio files, and now I can control the fades thanks to a video shader (“Fade”) and the audio gain parameter, respectively. So for each sequence I have 4 “tracks” : video, audio, nodal and automation. Looks good so far.

The main issue I see now is that the used RAM grows quickly, much beyond the loaded files size. With previous versions of score (v3 betas) I got up to 15 GB usage, now with first stable 3.0.0 it seems to behave better, but I had previously written an ofx OSCQuery-enabled video player that still shows even better performances than the latest release.

Another (not directly related) issue that keeps me hesitating about managing all medias from score is the “Images” process : I also need to load many images (about 1000) and show some of them that are randomly picked, for EEG calibration purpose. It seems that they are not loaded in memory from start, because when I reach the Images process in the scenario it freezes for a long time and the RAM also grows much beyond the loaded files size, and then I am able to show the images smoothly until I restart the scenario. For information the files take a little more than 300 MB, and the memory usage grows above 6 GB, which seems a bit weird.

So for now I think I will add this functionnality to my ofx program, which will also make it easier for me to display procedural visuals on top of the video sequences (I currently have no idea how to do this with shaders or custom score plugins), and use score to manage the timing/logic of the project only.

Just reporting the issues for this particular use case, in case it helps for the development.
Cheers !

PS : just forgot to mention another video related issue : when going into fullscreen, the video is stretched to fit the screen size, so the used projector dimensions MUST match the video dimensions otherwise the result is not what one would expect (black stripes to fill extra space without cropping the video instead of stretching)

Hi Joseph :slight_smile: I moved the message in a new post.

Thanks for the feedback !

now with first stable 3.0.0 it seems to behave better,

this is reassuring as I’ve worked on it before the release but yes still work to do

The main issue I see now is that the used RAM grows quickly, much beyond the loaded files size.

hmm, one thing is that score decompresses everything upon loading if it’s compressed (or for audio not at the correct sample rate). So e.g. if you have a 25 min mp3 or 48khz .wav loaded and the session is at 44100, it is converted to raw data when loading and kept in RAM which can explain the memory you see. I’ve been working on making the conversion “live” during execution but that’s not finished (and that eats more ressources when playing so I guess that will have to be a setting somewhere).
Same thing for images: they are decompressed and converted to RGBA when loading. And as you noticed it makes a lag when you have many loaded at the same time as they are all uploaded to the gpu when the image process starts. Fixing that requires some deeper changes (I had looked into it a few weeks ago because I encountered the same issues for some large JPEGs), but I’m going to add the issues, it will be fixed at some point !

PS : just forgot to mention another video related issue : when going into fullscreen, the video is stretched to fit the screen size, so the used projector dimensions MUST match the video dimensions otherwise the result is not what one would expect (black stripes to fill extra space without cropping the video instead of stretching)

do you think you could list the cropping / stretching options (and anything related) that would be useful for you here: Stretch / crop modes for videos & images · Issue #1346 · ossia/score · GitHub ?

Cheers and thanks again for the feedback, it’s super useful !

Thanks for your answer Jean-Michaël !

Actually I have the same issue with the pictures in my ofx application : the jpegs are decompressed and eat the same RAM (about 6 GB from a 330 MB set of images) as Score.
Actually my workaround is to pre-select a set of images and only load these (I don’t need to display 1182 images each time the program runs), but I didn’t find a way to do it from Score, as it must be done from a description text file.
I guess being able to load and parse the text file and lazy-load the pictures from a scripting process would be the best approach in my case.
I noticed a few other issues when loading a large amount of pictures in the Images process, I will open another thread …

Concerning the RAM usage with the videos, I think I should give another try playing the full scenario and see how much the RAM grows, and compare again with my other ofx application.
I’ll report back here with more precise data.

Cheers