Score as a Video Instrument for Live Performance

score
#1

Hi Everyone.
In search for an instrument on which I can set in-/outpoints in videos to loop them and create music on stage, I found ossia score, which looks very promising and might be the way to go (easier than coding everything in the terminal in realtime with python, ffmpeg, mpv or gstreamer).

However,
while using ossia score today, following questions appeared:

  • transform the video (position X,Y, scale)?

  • set the in/outpoint of the part to be looped and dynamically play with this numbers until it loops perfect?

  • whats going on with framerates? Openend a 50 fps testvideo and this was playing way to fast, is there currently any limitation?

  • how does the timing behave with videos, e.g. after setting “Loop Duration” to 4 . 0 . 0 . 00 it did not loop, but playing the video much faster after 4 seconds.

  • what does the “Enable Tempo” checkmark and the “Tempo” setting do?

  • any idea when direct audio support (video) or automatic extraction from the video file will be available?

Running on Ubuntu Linux, Score 3.1.0
very motivated :slight_smile:

#2

Hi and welcome !

(easier than coding everything in the terminal in realtime with python, ffmpeg, mpv or gstreamer).

that is certainly our goal :grin:

but right now honestly yes, the video object had some bugs and hasn’t been maintained enough

transform the video (position X,Y, scale)?

there’s a very basic “video mapper” object ; these last few weeks I’ve been working on actual 3D objects that will enable exactly this: Streamable

Can you share the file with the 50fps framerate ? (can be through wetransfer at contact@ossia.io) ; I thought that it worked but maybe a bug has cropped up since last time I checked.

how does the timing behave with videos, e.g. after setting “Loop Duration” to 4 . 0 . 0 . 00 it did not loop, but playing the video much faster after 4 seconds.

hmm, so this only really work with seekable videos, e.g. basic MJPEG avi or if you can, HAP. Otherwise seeking takes too much time in ffmpeg for, say, H.264 video and we loose the sync entirely ; haven’t yet found a proper fix for this short of reencoding the whole video in a properly seekable format when it’s loaded in score.

what does the “Enable Tempo” checkmark and the “Tempo” setting do?

basically it’s for videos which should be synced to the tempo of a music - if unchecked the video will always play at its builtin framerate without taking into consideration the speed adjustment that may occur if one adds a tempo curve automation to the music.

any idea when direct audio support (video) or automatic extraction from the video file will be available?

I’m finishing the 3D features and I’ll look into it - mostly it’s not yet there because I’m not entirely sure of the best way to present this to the user - modify the video process to show the attached soundtrack directly with it or instead create a separate video and sound process which is more flexible, but may make it easier to cause desyncs

#3

And - just to add: knowing that people are actually using the feature outside of our team is a very good motivation for fixing bugs in these areas so thanks for reporting them ! :slight_smile:

#4

Thanks for your quick response!
Here the video file which plays way to fast in ossia score: 01_dmx_tester.zip
It changes every second the color and the sound matches the color change as well, ideal for testing…

there’s a very basic “video mapper” object ; these last few weeks I’ve been working on actual 3D objects that will enable exactly this: Streamable

This looks interesting, I know how hard the whole mathematics on mapping stuff is, but why not just simple 2d: positionX,Y, scaleX[Y], would be very helpful to e.g. place 3 x 3 videos onto a HD screen…

  • so you are using ffmpeg? Maybe I can trigger it via a bash script from score? I am currently triggering my loops via terminal and mpv, simple things like loop in-/out, video position and size, like this:
    mpv "filename.mp4" --ab-loop-a=00:00:00 --ab-loop-b=00:00:00.100 --geometry=300x240+0+500 --no-border --speed=0.5 --profile=low-latency but stuck in simple bpm/sequencer triggering to get it repetitive to some musical time…

  • I think currently the “Enable Tempo” checkmark does nothing at all…

  • I would import the video/audio as one and also present it as one to the user, if the user needs a
    detached soundtrack, he can import a separate one at any time…

#5

hmm, so this only really work with seekable videos, e.g. basic MJPEG avi or if you can, HAP.

So I would have to convert all my videos first, similar to vidifold.com software.
Should it then work as expected…? Going for MJPEG?

Edit:

  • Converted video to HAP (ffmpeg -i input.mp4 -c:v hap output.mov)and loop is now working as expected.
  • But the enable tempo checkmark does nothing, does not matter if checked or not, video tempo will change while changing timeline tempo.

So I can go experimenting further, still believe this could be my new tool…

#6

Note that in the latest version finally the soundtracks are imported with video files automatically =) gotta check the tempo issues next…