Voice computing for special educational needs

The Council for Scientific and Industrial Research (CSIR) has developed an application (app) aimed at helping school learners who face barriers to reading and learning, as a result of early literacy difficulties, visual impairment, dyslexia and other special educational needs.

The app adds and synchronises human-narrated or computer-generated audio to text that is written in any of the 11 official languages of South Africa. “The app targets both the day-to-day content of educators and the textbooks of publishers. It adds value to the publishing industry by breaking down these barriers to turn non-readers into readers and increase the industry’s addressable market size,” says CSIR voice computing research group leader Karen Calteaux.

Named iSinkwe, which is the isiZulu word for a bushbaby, the app aims to allow users to create more accessible textbooks, class notes and other documents.

iSinkwe functions by adding audio, which is either recorded manually by a voice artist or generated automatically by a computer, to a standard Electronic Publication Version Three (EPUB3) document. It automates the synchronisation of the audio to the text of the document at word, sentence and paragraph levels, allowing users to switch between these levels when they read or navigate through the document using audio and highlighting. It does not translate the content from one language to another, but simply adds audio to documents that are already written in the language of the user’s choice.

“If users only have a Microsoft DOCX or PDF document, they can convert those documents automatically to the standard EPUB3 format using iSinkwe Convert. They simply upload the document to the iSinkwe website, and it will provide them with an EPUB3 version that they can download,” says product leader Georg Schlünz.

iSinkwe Synchronise allows users to upload a standard EPUB3 document and the pre-existing human-narrated audio files, to the iSinkwe website. When no human-narrated audio is available, computer-generated audio is added instead. It also allows for a combination of human-narrated and computer-generated audio. Users can then download the EPUB3 document with the synchronised text and audio included.

To read any of the EPUB3 documents with synchronised text and audio, users need to install the iSinkwe Read app, which is currently available for Android, with iOS and Windows versions coming soon. Once the EPUB3 document is added to the library and opened, users can listen to and navigate through the audio on word, sentence and paragraph levels. Text highlighting on these levels follows the audio automatically as it is played. Users can customise the reading experience further by changing the font type, font size and background colour, as well as the playback speed of the audio and associated text scrolling behaviour.

iSinkwe is now available for licensing by resellers.