356-winter-2024/hw3

Programming Etude #3: "Wekinate Your World"

Music and AI (Music356/CS470) | Winter 2024 | Ge Wang

In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is good! Playful is wonderful!

Due Dates

• Milestone: webpage with 2 initial experiments Monday (2/19, 11:59pm)
• Final Deliverable: due Monday (2/26, 11:59pm)
• In-class Presentation: Tuesday (2/27)

Discord Is Our Friend

• direct any questions, rumination, outputs/interesting mistakes to our class Discord

Things to Think With

• some relevant readings (including from your Critical Response #2 assignment)
  • "Power to the People: The Role of Humans in Interactive Machine Learning"
  • "Humans in the Loop: The Design of Interactive AI Systems"
  • Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance

Tools to Play With

• get and play with Wekinator
• get the latest [https://chuck.stanford.edu/ (version 1.5.2.0 or higher)
  • ChAI and API reference

Getting Started with Wekinator

• a terrific online course on Kadenze from Dr. Rebecca Fiebrink on interactive ML for music using Wekinator:
  • "Machine Learning for Musicians and Artists"
  • in particular, the videos for the following sessions can be particularly helpful for this assignment:
    • Session 5: Sensors and Features: Generating Useful Inputs for Machine Learning
    • Session 6: Working with Time
    • Session 7: Session 7: Developing a Machine Learning Practice; Wrap-up
  • These videos are clear, concise, and practical! We highly recommend taking a look as you explore Wekinator and this homework!
  • NOTE: to view these videos, you will need to 1) sign up for a Kadenze account, 2) upgrade for the (free) 7-day premium membership trial; for anyone wanting to reference the course beyond the trial, Ge will reimbursement one-month of premium membership.
• Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms:
  • (primary approach) the Wekinator application (GUI) is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the Open Sound Control (OSC) communication protocol
    • the system architecture looks like INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC:
    • INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models
    • OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm)
    • in this approach, Wekinator app's TRAIN mode iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's RUN mode predicts output based on new input
  • (secondary approach) the Wekinator object in ChucK allows you to work with Wekinator using code and without the need for network communication
    • Wekinator can TRAIN (from input-output observations) and/or RUN (predict output from new input)
    • or use Wekinator.loadData() to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved using Wekinator.saveData()
    • helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the Wekinator app, save the data; (step 2) load the data in the Wekinator object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC)
• for this assignment, we recommend trying both before deciding what to use for each of your three system designs
• Resources for learning and working with Wekinator
  • check out some example projects people have created using Wekinator
  • for a general overview of how Wekinator works, here are basic instructions
  • get started with using Wekinator walkthrough
  • for a more detailed reference, check out Wekinator detailed instructions
  • example code for working with Wekinator app, OSC, and ChucK
  • example code for working with the Wekinator object within ChucK
  • here are a trove of examples for working with Wekinator across many languages

Your Task

• observe and identify three activities or tasks you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small)
• of your three systems, at least one should be an interactive and expressive musical instrument designed with Wekinator for regression
• experiment (a lot) with different ideas (possibly more than the three you end up with)!
• once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are virtues here
• deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems!
  • "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..."
• for each system, create a short video showing the system in action

A Few Bad Ideas!

• Using camera input and sound output, create a day / night classifier (or sonifier)—I mean, who has time to look out the window?
• An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—try playing this for your roommate, or your cat (and film the performance and audience reception)
• using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice or gesture command system that does meaningless things—inject a little AI-mediated chaos into your day
• Use Wekinator to train an instrument for controlling/exploring your audio mosaics (the output would be an N-dimensional/dimensionally-reduced vector)
• for more ideas, check out these projects (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)!

Reflections

• write ~300 words of reflection on your etude. It can be about your process or the products. Tell us about your attempt to deploy them.

Final Deliverables

• create a CCRMA webpage for this etude
  • the URL should live at https://ccrma.stanford.edu/~YOURUSERID/356/hw3 or https://ccrma.stanford.edu/~YOURUSERID/470/hw3
  • alternately, you may use Medium or another publishing platform (but please still link to that page from your CCRMA webpage)
• your webpage is to include
  • a title and description of what you made (free free to link to this wiki page)
  • three video recordings corresponding to your three interactive AI systems in action
  • all relevant code and data files for all three systems
  • brief report on how you created these
  • your 300-word reflection
  • any acknowledgements (people, code, or other things that helped you through this)
• submit to Canvas only your webpage URL