Conversational Internet

tl;dr

We’ve built a prototype to show how we could interact with the Internet using a command-driven approach.

  • A screen reader, but one that uses machine learning and natural language processing, in order to better understand both what the user wants to do, and what the web page says.
  • One that can offer a conversational interface instead of just reading out everything on the page.

It’s a proof-of-concept, but it’s an exciting idea with a lot of potential and we’ve got a demo that shows it in action.

The problem : screen readers today

I’ve written about this before but here is a recap.

Visually impaired people can interact with the web using screen readers. These read out every element on a page.

The user has to make a mental model of the structure of the page as it’s read out, and keep this in their head as they arrow-key around the page.

For example, on a news site’s front page, once the screen reader has read out the page, you have to remember if the story you want is the fifth or sixth story in the list so you can tab the right number of times to get to it.

Imagine an automated telephone menu:
“for blah-blah-blah, press 1, for blather-blather-blather, press 2, for something-or-other, press 3 … for something-else-vague, press 9 …”

Imagine this menu was so long it took 15 minutes or more to read.

Imagine none of the options are an exact match for what you want. But by the time you get to the end, you can’t remember whether the closest match was the third or fourth, or fiftieth option.

The vision : a Conversational Internet

Software could be smarter.

If it understood more about the web page, it could describe it at a higher, task-oriented level. It could read out the relevant bits, instead of everything.

If it understood more about what the user wants to do, the user could just say that, instead of working out the manual navigation steps themselves.

The vision is software that can interpret web pages and offer a conversational interface to web browsing.

Continue reading

Hursley Extreme Blue 2011 Presentations

Extreme Blue logoExtreme Blue is IBM’s summer intern scheme. Students can apply to IBM to be part of the scheme and those lucky enough to be selected are brought into various IBM locations worldwide to be mentored by IBM staff who have proposed an idea and small project for them to work on.

This morning I went along to listen to what the 16 students in the UK have been doing with their summer. These students were split into four groups of four, working on projects for an improved voting system, a smart cursor, smarter vehicles and FTP discovery.

You know you’re getting old when all the students seem rather young, I think “green” is the term people used to use when I was starting in IBM, they do remind me of my early days at work. However, they all presented themselves beautifully, spoke very well using slick rehearsed presentations they’ve put a lot of effort into, and (barring one or two stutters) seemed entirely confident in what they were doing up at the front of what must seem an intimidating auditorium full of knowledgeable IBM professionals. They handled questions well too, I don’t necessarily have to agree with all the answers, but the way they each went about receiving the questions and providing thoughtful answers was good.

Each team had 7 minutes to present their 12 weeks’ work with every person in the team getting a chance to pitch in at some point, so they didn’t get very long to put their projects across. The audience were asked to keep questions until the end of the pitch, which allowed them to flow easily through their material. The range of presentations was interesting, some chose to manually click through PowerPoint-style, while other groups came up with stories or monologuing through a video they had created. This range kept the audience interested with each style of presentation being effective for its purpose.

It was interesting to see how each of the projects has been clearly influenced by the four members of the team. Each team of four contained one business student and three technical students, and the range of skills came through in the presentations. Some groups had “deep-dived” straight into technical work while others had spent more time thinking about use cases, business cases, how their project might fit in with IBM or be sold. I suspect this has a direct relationship to both how the team was lead by the IBM staff but also by the particular characters of each team and reminded me of Myers-Briggs or Belbin style studies I’ve done in the past.

Now I’ll have a little look at each project in (very) brief… I’ll stress in advance that I’ve heard a small snippet of 12 weeks of hard work and any opinion here is mine alone and based solely on today’s pitches:

Improved voting system
The team gave an introduction to their solution involving a three phase voting system followed by an example of the problem they were trying to solve and how their solution tackled these. The team had been working with a local council to identify requirements for such a system, so were able to work with real-world examples and solicit feedback. Questions followed and feedback from the council seemed to have been good. Some doubts were expressed by the audience about the security of such a system which whilst possibly valid, it seemed to me that these could be addressed should the solution be implemented live. The team presented the solution as having environmental benefits which might seem obvious at first but I thought were rather questionable given the requirement to use computer hardware and power, a further study would be required here to determine whether the current system using sustainably-sourced paper could be bettered on the environmental front. Verification of voters appears to be vastly improved using their system with less room from fraudulent votes with connection to other systems for authentication such as the DVLA. Clearly any such automated voting system would have huge benefits for the speed of counting after voting has completed.

Smart Cursor
A new input device to control an on screen cursor using any sort of body movement aimed at improving human-computer interaction (primarily for disabled people). The system involves a hardware sensor strapped to the part of the body that has movement. Initial calibration for any new part of the body is required which is run once to set up 4 movements (up/down/left/right). Other movements and gestures would also be possible such as a mouse click and the combination of sensors on multiple parts of the body. The hardware technology could be built small enough to be permanently wearable without distress or difficulty to the user. Other uses of the technology appear to be for rehabilitation or monitoring a condition whilst wearing the hardware device. Lots of room for customisation brought out during questioning as well as a few issues about how to set up the device in the first place. However, this seemed like a really worthwhile (if low usage) piece of research that could be immensely useful to its target audience and at low cost too.

Smarter Vehicles
The aim of this project is to personalise the driving experience for car users by attempting to add three things to a car (1) identifying which user is driving, (2) providing the car with knowledge about where it’s going, and (3) permanently connecting the car to a network. The team used a video style presentation and monologue they had story-boarded which was clearly well produced and rehearsed. It was unclear what the project had achieved, however, as no specifics were mentioned on what had been achieved but there were certainly plenty of good ideas as to what could be done in this area. The team do appear to have a demonstration available which I’m looking forward to going to see in Hursley tomorrow and the Extreme Blue demonstration expo after which I’m sure it will be a lot clearer which ideas they’ve followed through into something tangible and which are still in progress. Another great plus for this team was they were aligned with an automotive manufacturer and will be presenting their ideas back to the board at a later date which will be a fabulous experience to get for them all.

FTP Discovery
Tackles the problem of escalating FTP network complexity in enterprises. The project attempts to map FTP files on the network in flight and automatically provides a visualisation of the network in a node graph style format. This network can be annotated manually with things such adding the cost of various transfers and links to allow the users to build up a visual picture and cost to the company of their FTP services. The team advocate the use of managed file transfers (as provided by WMQ File Transfer Edition, for example) but failed to clearly state what the problem with FTP as a service is. That said, they seem to have a very clever way of detecting FTP traffic by sniffing the network and could easily extend their architecture to include all sorts of other protocols. They have also thought carefully about how their work might be used in the future, for example as a tool for IBM pre-sales, a saleable IBM product or (most likely) a component of one or more existing IBM products.

Congratulations to all the teams and people involved. The presentations were great, a very entertaining hour, and it seems like some really useful work has come out of Extreme Blue in the UK again this year. Well done!

The Eightbar brand – part 3

Following on from Ian’s posts about the Eightbar gang sign and large hands and sign language in Second Life I have created the Eightbar gang sign using my good friend Anna.

Anna is an animated avatar developed by the University of East Anglia’s eSign project to synthesize sign language, she was used as part of an Extreme Blue project to convert Speech to Sign last summer. Anna is animated using Signing Gesture Mark up Language (SiGML) which is based on the internationally established notation for sign, HamNoSys. Currently, to create signs for Anna, eSign have provided an editor which is very good, but requires a reasonable amount of time to be able to use efficiently. For a few days per week I have been working on a way for people who are not familiar with the eSign editor to create signs for Anna, with the hope that creating signs can be the sort of thing you just dip into, when you have a spare minute and a sign you would like to create. With this in mind I decided, as a test, to create the EightBar gang sign using my interface.

After 15 minutes of playing about, I had a gang sign! It took a few attempts to get there mind…

But finally Anna was throwing the gang sign like a pro…

Eightbar gang sign