Project

Seeing Sound is a collaboration between two campuses at LUCA School of Arts and aims to stimulate the equal, interdisciplinary collaboration between music composers and filmmakers. A case-based research project was carried out in the period September 2012 – September 2014, which included:

• Organisation of a course module that could be taken by students in music composition (3rd Bachelor, campus Lemmens) and animation film (3rd Bachelor, campus C-mine). The teachers of the course were composer Jeroen D’hoe and filmmaker Luc Leon Smeets.
• Collaborations between students in the context of the graduation projects of master students in Animation Film.

Research committee

Both courses were monitored by a team of researchers of different departments in the overarching Association of KU Leuven University. The following researchers have actively participated, on a weekly basis, in the coaching of students and have reported the main results:

• composer Hans Helsen (LUCA School of Arts, campus Lemmens)
• filmmaker Dirk Henrotay and researcher Steven Malliet (LUCA School of Arts, campus C-mine)

In addition, the following researchers have monitored the project and provided technical and pedagogical feedback on a regular base:

• Bas Bergervoet, Erwin Heylen and Roman Verraest (KU Leuven, LIMEL)
• Jan Elen (KU Leuven, Department of Instructional Psychology and Technology)
• Carl Van Eyndhoven (LUCA School of Arts, campus Lemmens)

Although the project was officially terminated in September 2014, both courses continue to be organised, with Jeroen Dhoe, Dirk Henrotay and Steven Malliet as teachers. Accordingly the results, as reported on this site, are continuously being refined and updated.

Research questions

On a conceptual level, we have formulated the following research question:

Which mutual vocabulary can music composers and filmmakers use to collaborate in an equal, co-creative way?

At the level of production, we have formulated the following research question:

How can this mutual vocabulary be used to create a range of innovative short movies?

As regards the organisation of a course module the following research question was added:

Which educational strategies are most suited to organise a multi-campus interdisciplinary module, involving students in music composition and animation film production?

Methods

Synesthesia

Concerning the organisation of the courses we have adopted a synesthetic approach, whereby music composers are introduced to the jargon of animation filmmakers, and vice versa. Each year, a number of teams are made, consisting of at least one student in music composition and at least one student in animation film. In the conceptual phase students are stimulated to use the synesthetic perspective to create innovative ideas. In the production phase students implement the score and visuals of a short movie, providing each other with feedback on a weekly base.

Participant observation

The production process has been closely monitored by 3 researchers. Participant observation was used a main research method. The researchers, who are specialized in music composition, animation film production and design research, have participated on a weekly basis in the coaching of students, and have closely observed the interactions between students. Additionally, on finishing the module after 8 weeks, students have been interviewed in groups of 2 or 3 (as they had collaborated). These interviews can be considered a hybrid between a face-to-face interview and a focus group interview. The interviews have been structured around the field notes the researchers have made throughout the year, which had been coded and analysed iteratively. Finally, document analysis has been used to examine the interactions between students, artefacts that were created, and student reports.

Communication platform

The researchers of LIMEL have helped in setting up a communication platform that facilitates the transfer of (audio, image and video) files, the interpersonal communication and the exchange of ideas and artistic accomplishments. Based on this a functional description has been made of a software layer that can later be implemented as middleware between music composition software and editing software.

Results

• Films. The films that have been made in the period 2012-2014 can be found on this site.
• Teaching. An overview of the educational format we have developed can be found on this site.
• Vocabulary. A documentation of the intermediary vocabulary can be found on this site.

These results can, in a follow-up project, be used to create a software layer that operates as middleware between exisiting music composition programs and existing editing software. The functional description of this software layer consists of two documents:

• A technical document describing the affordances of the software
• An interactive mockup demonstrating the functions of the system

The tool should facilitate the creation of a storyboard/score (as explained in the class on this topic) by implementing the following aspects:

• It should be structured around the creation of a timeline, which can be represented in terms of time indicators (seconds and minutes), or in terms of music bars.
• It should allow users to provide a background layout, against which the timeline is displayed. This could be used to indicate a general transition of mood or atmosphere
• It should be possible to map different nodes of information (verbal, acoustic, visual, audio-visual, symbolic) onto specific time moments
• It should allow users to expand or shorten the timeline without losing information
• It should allow users to proceed in two phases: first make a mapping of sources of inspiration, and second make the transition to the creation of a storyboard/score