User:Balachandarkrishna

This web page is to track my work progress of the ongoing auditory Graph project at Sonification Lab at Georgia Tech

First Week (Aug 31 2009 - Sep 4 2009)

Export Menuoptions in Sandbox

Provide Menu options namely export with sub menu option namely MIDI,PNG and CSV , these options would enable a user to export his work to a external file on the local disk. This makes use of the available Reifier classes to export the appropriate contents to the file.

Progress Monitor for export options

Provide a progress monitor feature to enable the user to track the progress of the export option. This is particularly useful when the export command takes a longer duration than normal. This is implemented through ProgressMonitor class in Java and use of the TranslationMessageListeners in the sandbox Model.

--Balachandarkrishna (talk) 14:00, 7 September 2009 (UTC)

Second week (SEP 8 2009 - SEP 13 2009)

NASA Math Description Engine.[DEMO Preperation]

I was able to use the MDE and genrate text descriptions for graphs, MDE offers a good textual description and is really powerful for math functionality, itwas able to handle almost any type of curve given a schematic equation in accordance with the parametric equation of the curve, MDE accurately generates graphs for it.

I have demo ready. One feature lacking in MDE is the sonification part, it seems pretty inadequate. Importing MDE into our sandbox will provide lot of functionality for the sandbox.

Bug Fixes

Progress Monitor not visible

Help menu - Help Menu is present, when clicked on the Help contents, it throws null pointer exception due to no help set being present. This could be a bug which could be discovered easily by a user. - There is also a Help Button present in the Sandbox UI at the right bottom, this can be removed, since help men is already available, this button when clicked throws exception for helper set. - Disable help for now - don't provide any UI links to it. The content is not up-to-date.

Text boxes for project information - There are two text boxes in the Sandbox UI for project name and Author name, they don’t Seem to convey any useful information. Either they can removed or we can display currently Selected project in project name. Author of the project can be removed. - these could be used later, but aren't currently being stored in the model. Just remove these text boxes.

Sonification Sandbox 6.0.0 beta .splash screen is v5 .app title in title bar of window and menu is v5 Fixed .

auto-played theme in the splash screen..in fact, I know it is at least too loud, and probably should not auto-play ---> Play button introduced.

File..Examples is empty, but menu is still active ( Examples temporarily Disabled)

The Help button does nothing. Should be disabled if no help is available. Should probably at least have a stub help in there, with a URL, some basic FAQs, or whatever.

The Export button works, leads to the Export dialog box: Could use some help in this dialog for the settings. Like what is Complete versus Simple in the Visual Graph? Fixed by BALA sep-13-09 , Redundant fuctionality between export options and export menu as per discussion .

The export process did not seem to work, despite the message that said it was "ready to export" {Fixed as a part of export menu, export menu provdes these options} Balachandarkrishna (talk) 01:25, 14 September 2009 (UTC)

Week 3: September 15-september 21 2009

Implementations

NASA MDE DEMO and FreeTTS Text to speech

Demo of the prototype app built on top of NASA MDE to showcase its functions. FreeTTS library is being considered to be used as a text to speech converter

Development of Auditory Graph application for blackberry

This is the main one for the week. This involves developing a new application for the blackberry smartphone. The application does the following

a) Capable of Playing a Midi file. Use the Blackberry Player APIS to do this.

b) Develop a Menu which will list all music items available. This involves dynamic generation of menu items dependent on the contents of the directory.

c) The SDCARD of the Blackberry smart phone was simulated through the use of the local pc file system.

d) Mapping of the Dynamic menu items to the resource files (.midi). The files which needs to be played when the menu is clicked.

e)Displaying the corresponding graphs as a PNG file when the audio is playing, this involved application module communication and posting a event to the UI APP moadule. This module will then create a new screen which is used to display the image file.

BUG Fixes:

This week mainly dealt with development work. However a important bug fix was added to AUditory Graph Model.

Refinements to the play toolbar a) Implementation of loop functionality, this feature will make sure that audio loops infinitely until the user turns the loop off again by clicking the loop button.

b) Indentation and visual cues for the user to identify which is the current button clicked or which one is operational in the play tool bar.

September 21-October 17

All Developments in sandbox and AGM

Bug Fixes

In the Pan area, there are some characters that do not display correctly. I see: "Default: 1? left...." where ? is a non-displaying character, and right below it, the "Min: 90?" and "Max: 89?" fail to display characters

In the Visual graph settings, the line color is automatic, and shows as black, but in the actual visual graph, the line and symbols are red. When you select

"Select Color" and choose, say, blue, the sample line to the right changes, but the actual graph on the Visual Graph tab does not change.

When you change the Stroke Type, the sample line does not change (nor does the actual graph).

Changing the Line Color back to automatic changes the sample line back to solid black, but does not update the graph.

And, right below the Time section, there is a check box and a text label "Name: See above. --Oh, I see that if I enter "Bruce" in the box, it renames the data column and the legend...but not clear or intuitive.

The "View Mappings" button needs to change dynamically to "Hide Mapping" and "Show Mapping" as appropriate (not View Mappings, and especially not View Mappings when the mapping is already displayed)

What is the "Type" dropdown for? Should it be renames "Data Type"?

In the Pitch area, again, what does the checkbox do? It is a control with no label, and seems to be associated with Timbre. Same goes for Pan and Volume.

The Export button works, leads to the Export dialog box:

Could use some help in this dialog for the settings. Like what is Complete versus Simple in the Visual Graph? Fixed by BALA sep-13-09 , Redundant fuctionality between export options and export menu as per discussion.

When I change the data column name in the Mappings tab (see below), I see that it changes in the Visual Graph legend, but there seems to be a leading "0 " (I set it to "Bruce" and see "0 Bruce" in the legend.

MDE and Free TTS Integration into sandbox

Importing MDE into our sandbox will provide lot of functionality for the sandbox. MDE provides textual description for graphs.

MDE has been successfully integrated into the sandbox and demoed at GVU and is a part of sonification sandbox beta Release.

The idea is to use the FreeTTS text to speech coversion Java library along with the textual descriptions for graphs provided by the NASA MDE . This feature can be integrated into the sandbox to provide a comprehensive tools for the AGM.

The Free TTS and AGM has been successfully integrated into the sandbox and used in GVU Demo. Free TTS will not be a part of Sonification sandbox version 6.0 Beta Release.

Formula Bar addition to Sandbox

New formula bar support has been added to sonification sandbox 6.0. The feature allows user to derive data series based on existing data series. A new Recursive expander has been added.

Currently formula bar does not expand functions like SIN and COS. This is targeted for the main release of sandbox 6.0.0

Order of evaluation of expressions is preserved and the formula expanded interprets any data series equation expressed in term of columns and numbers.

GVU Demo

The latest version of sandbox 6.0 and AGM with latest features were demo ed in School of Interactive computings Demo on October 15 2009.

New Reifier in Auditory Graph Model

A new Reifier for describing a Graph named as the chart describer Reifier has been added to the AGM. This provides support for text descriptions of the graph using MDE.

October 18-November 14

Internal Beta relese version 1, 2 and 3 have been released

Sandbox Animation Export

The animation feature has been built into the sandbox. this is done using the Java Media Framework. The first phase of the implementation is

a) Enabling a Line moving across the graph indicating the point sonified. There was a issue with th event system, leading to non determinism in the inter time arrival. This led to lag in the Graph Marker played across the graph. This was fixed by the Error estimator which compares the expected and actual position and making error adjustments as needed.

b) A Live Recording feature was designed for sandbox, this uses the Robot class of the awt to generate screen captures and string these screen captures as a movie. This is inefficient and a better way will be to predictvely place the graph marker on the image during play out. Large amounts of spatial redundancy is present in this images which can be exploited for space savings.

c)The generated AVI/Quick time is stored in the Sandbox Jars current path.

Global Mapping scales for Pitch Pan and Volume.

One of the important features for sandbox 6.0 is the global mapping scales for pitch/pan/volume. When two Lines of the same proportion (say multiple lines are in a graph and both lines exhibit the same parametric curve albeit on different scales). The visual graph clearly indicates this on the screen. To capture this information in the MIDI, there needs to be a notion of global maximum and minimum, this will help in representing multiple lines are present in the graph and their relative magnitudes. Before this feature both the graphs wil sound exactly alike.

Sound sweep from Left to Right and Right to Left

New options were added to make the sound Pan from Left to right and vice versa. The Panning was added to the User interface as a sweep option in the Mappings panel. The default setting is fixed pan setting. The sweep option pans the sound from left to right. If the polarity is set as negative and the sweep option was enabled sound pans from Right to left.

New Light weight Save feature

The entire serialization process of saving the project to disk was resulting in number of issues because of non serializable objects.This was taken care by creating a minature version of the graph object before attempting persistence to the disk. This light weight object is now totally independent of the members of the graph objects and it is easy to add new attributes to graph object without sacrificing persistence.

Individual Line descriptions

A new feature was added which allowed describing each line in the graph individually and also the entire graph as a whole. This finer level of granularity aids the user to inspect only those dependent series which are of interestto him.

CSV File Import into the sandbox

This feature was not functioning previously in sandbox. Tnis was fixed and now CSV files can be imported into the sandbox. There was an issue with loading a new auditory graph project associated with the new data series values present in the CSV. This was fixed and now on loading the new CSV, the new project automatically takes effect.

UI Revisions to the sandbox

Sandbox UI was totally revamped to support

a) Tabbed Mapping panel.

b) Intuitive labels and controls

c) Uniform Name association for Data series across panel

d) Refining formula BAR to enhance usability.

e) Refining spread sheet panel for usability.

f) Export wizard sync with file menu and cleaning archaic options in export Menu.

Visual Rendering setting for a line

When the Lines Mapping is turned off, MIDI Rendering for the line does not take place, however the line is still displayed on the Graph. A new feature was added to control the visual graph rendering on the screen and the ability to selectively turn lines either on or off.JFreechart Reification process was modified to achieve this.

Video Capture Project

Using JMF a new project wa created named Video capture project which allows generation of AVI/Quick time Movies and also allows conversion from MIDI to Wav File.

a) Video capture project allows creation of AVI stream from set of Buffered Images.

b) Ability to set frame rate and control duration of movies.

c) convert MIDI to WAV file format

d) Merging multiple audio/video streams into a single source stream with Audio/Video interleaving