A Conversational Internet of Things – ThingMonk talk

Earlier this year, Tom Coates wrote a blog post about his session at this year’s O’Reilly Foo Camp. Over tea with colleagues, we talked about some of the ideas from the post and how some of our research work might be interesting when applied to them.

One thing led to another and I found myself talking about it at ThingMonk this year. What follows is a slightly expanded version of my talk.


ciot-0

ciot-2

Humanising Things

We have a traditional of putting human faces on things. Whether it’s literally seeing faces on the Things in our everyday lives, such as the drunk octopus spoiling for a fight, or possibly the most scary drain pipe ever.

Equally, we have a tendency to put a human persona onto things. The advent of Twitter brought an onslaught of Things coming online. It seemingly isn’t possible for me to do a talk without at least a fleeting mention of Andy Standford-Clark’s twittering ferries; where regular updates are provided for where each ferry is.

ciot-5

One of the earliest Things on Twitter was Tower Bridge. Tom Armitage, who was working near to the bridge at the time, wrote some code that grabbed the schedule for the bridge opening and closing times, and created the account to relay that information.

ciot-6

One key difference between the ferries and the bridge is that the ferries are just relaying information, a timestamp and a position, whereas the bridge is speaking to us in the first-person. This small difference immediately begins to bring a more human side to the account.
But ultimately, they are simple accounts that relay their state with whomever is following them.

This sort of thing seems to have caught on particularly with the various space agencies. We no longer appear able to send a robot to Mars, or land a probe on a comet without an accompanying twitter account bringing character to the events.

ciot-7

There’s always a sense of excitement when these inanimate objects start to have a conversation with one another. The conversations between the philae lander and its orbiter were particularly touching as they waved goodbye to one another. Imagine, the lander, which was launched into space years before Twitter existed, chose to use its last few milliamps of power to send a final goodbye.

ciot-8

But of course as soon as you peek behind the curtain, you see someone running Tweetdeck, logged in and typing away. I was watching the live stream as the ESA team were nervously awaiting to hear from philae. And I noticed the guy in the foreground, not focused on the instrumentation as his colleagues were, but rather concentrating on his phone. Was he the main behind the curtain, preparing Philae’s first tweet from the surface? Probably not, but for the purposes of this talk, let’s pretend he was.

ciot-9

The idea of giving Things a human personality isn’t a new idea. There is a wealth of rigorous scientific research in this area.

One esteemed academic, Douglas Adams, tells us about the work done by the The Sirius Cybernetics Corporation, who invented a concept called Genuine People Personalities (“GPP”) which imbue their products with intelligence and emotion.

He writes:

Thus not only do doors open and close, but they thank their users for using them, or sigh with the satisfaction of a job well done. Other examples of Sirius Cybernetics Corporation’s record with sentient technology include an armada of neurotic elevators, hyperactive ships’ computers and perhaps most famously of all, Marvin the Paranoid Android. Marvin is a prototype for the GPP feature, and his depression and “terrible pain in all the diodes down his left side” are due to unresolved flaws in his programming.

In a related field, we have the Talkie Toaster created by Crapola, Inc and seen aboard Red Dwarf. The novelty kitchen appliance was, on top of being defective, only designed to provide light conversation at breakfast time, and as such it was totally single-minded and tried to steer every conversation to the subject of toast.

ciot-13

Seam[less|ful]ness

In this era of the Internet of Things, we talk about a future where our homes and workplaces are full of connected devices, sharing their data, making decisions, collaborating to make our lives ‘better’.

Whilst there are people who celebrate this invisible ubiquity and utility of computing, the reality is going to much more messy.

Mark Weiser, Chief Scientist at Xerox PARC, coined the term “ubiquitous computing” in 1988.

Ubiquitous computing names the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives.

Discussion of Ubiquitous Computing often celebrated the idea of seamless experiences between the various devices occupying our lives. But in reality, Mark Weiser advocated for the opposite; that seamlessness was undesirable and self-defeating attribute of such a system.

He preferred a vision of “Seamfulness, with beautiful seams”

ciot-15

The desire to present a single view of the system, with no joins, is an unrealistic aspiration in the face of the cold realities of wifi connectivity, battery life, system reliability and whether the Cloud is currently turned on.

Presenting a user with a completely monolithic system gives them no opportunity to connect with and begin to understand the constituent parts. That is not it say this information is needed to all users all of the time. But there is clearly utility to some users some of the time.

When you come home from work and the house is cold, what went wrong? Did the thermostat in the living room break and decide it was the right temperature already? Did the message from the working thermostat fail to get to the boiler? Is the boiler broken? Did you forgot to cancel the entry in your calendar saying you’d be late home that day?

Without some appreciation of the moving parts in a system, how can a user feel any ownership or empowerment when something goes wrong with it. Or worse yet, how can they avoid feeling anything other than intimidated by this monolithic system that simply says “I’m Sorry Dave, I’m afraid I can’t do that”.

Tom Armitage wrote up his talk from Web Directions South and published it earlier this week, just as I was writing this talk. He covers a lot of what I’m talking about here so much more eloquently than I am – go read it. One piece his post pointed me at that I hadn’t seen was Techcrunch’s recent review of August’s Smart Lock.

ciot-16

Tom picked out some choice quotes from the review which I’ll share here:

“…much of the utility of the lock was negated by the fact that I have roommates and not all of them were willing or able to download the app to test it out with me […] My dream of using Auto-Unlock was stymied basically because my roommates are luddites.”

“Every now and then it didn’t recognize my phone as I approached the door.”

“There was also one late night when a stranger opened the door and walked into the house when August should have auto-locked the door.”

This is the reason for having beautiful seams; seams help you understand the edges of a devices sphere of interaction, but should not be so big to trip you up. Many similar issues exists with IP connected light bulbs. When I need to remember which app to launch on my phone depending on which room I’m walking into, and which bulbs happen to be in there, the seams have gotten too big.

In a recent blog post, Tom Coates wrote about the idea of a chatroom for the house – go read it.

Much like a conference might have a chatroom, so might a home. And it might be a space that you could duck into as you pleased to see what was going on. By turning the responses into human language you could make the actions of the objects less inscrutable and difficult to understand.

ciot-17

This echoes back to the world of Twitter accounts for Things. But rather than them being one-sided conversations presenting raw data in a more consumable form, or Wizard-of-Oz style man-behind-the-curtain accounts, a chatroom is a space where the conversation can flow both ways; both between the owner and their devices, but also between the devices themselves.

What might it take to turn such a chatroom into a reality?

ciot-18

Getting Things Talking

Getting Things connected is no easy task.

We’re still in the early days of the protocol wars.

Whilst I have to declare allegiance to the now international OASIS standard MQTT, I’m certainly not someone who thinks one protocol will rule them all. It pains me whenever I see people make those sorts of claims. But that’s a talk for a different day.

Whatever your protocol of choice, there are an emerging core set that seem to be the more commonly talked about. Each with its strengths and weaknesses. Each with its backers and detractors.

ciot-19

What (mostly) everyone agrees on is the need for more than just efficient protocols for the Things to communicate by. A protocol is like a telephone line. It’s great that you and I have agreed on the same standards so when I dial this number, you answer. But what do we say to each other once we’re connected? A common protocol does not mean I understand what you’re trying to say to me.

And thus began the IoT meta-model war.

There certainly a lot of interesting work being done in this area.

For example, HyperCat, a consortium of companies coming out of a Technology Strategy Board funded Demonstrator project in the last year or so.

ciot-21

HyperCat is an open, lightweight JSON-based hypermedia catalogue format for exposing collections of URIs. Each HyperCat catalogue may expose any number of URIs, each with any number of RDF-like triple statements about it. HyperCat is simple to work with and allows developers to publish linked-data descriptions of resources.

URIs are great. The web is made of them and they are well understood. At least, they are well understood by machines. What we’re lacking is the human view of this world. How can this well-formed, neatly indented JSON be meaningful or helpful to the user who is trying to understand what is happening.

This is by no means a criticism of HyperCat, or any of the other efforts to create models of the IoT. They are simply trying to solve a different set of problems to the ones I’m talking about today.

ciot-23

Talking to Computers

We live in an age where the talking to computers is becoming less the reserve of science fiction.

Siri, OK Google, Cortana all exist as ways to interact with the devices in your pocket. My four year old son walks up to me when I have my phone out and says: “OK Google, show me a picture of the Octonauts” and takes over my phone without even having to touch it. To him, as to me, voice control is still a novelty. But I wonder what his 6 month old sister will find to be the intuitive way of interacting with devices in a few years time.

The challenge of Natural Language Parsing, NLP, is one of the big challenges in Computer Science. Correctly identifying the words being spoken is relatively well solved. But understanding what those words mean, what intent they try to convey, is still a hard thing to do.

To answer the question “Which bat is your favourite?” without any context is hard to do. Are we talking to a sportsman with their proud collection of cricket bats? Is it the zoo keeper with their colony of winged animals. Or perhaps a comic book fan being asked to chose between George Clooney and Val Kilmer.

Context is also key when you want to hold a conversation. The English language is riddled with ambiguity. Our brains are constantly filling in gaps, making theories and assertions over what the other person is saying. The spoken word also presents its own challenges over the written word.

ciot-28

“Hu was the premiere of China until 2012″

When said aloud, you don’t know if I’ve asked you a question or stated a fact. When written down, it is much clearer.

ciot-29

In their emerging technology report for 2014, Gartner put the Internet of Things at the peak of inflated expectation. But if you look closely at the curve, up at the peak, right next to IoT, is NLP Question Answering. If this was a different talk, I’d tell you all about how IBM Watson is solving those challenges. But this isn’t that talk.

ciot-30

A Conversational Internet of Things

To side step a lot of the challenges of NLP, one area of research we’re involved with is that of Controlled Natural Language and in particular, Controlled English.

CE is designed to be readable by a native English speaker whilst representing information in a structured and unambiguous form. It is structured by following a simple but fully defined syntax, which may be parsed by a computer system.

It is unambiguous by using only words that are defined as part of a conceptual model.

CE serves as a language that is both understandable by human and computer system – which allows them to communicate.

For example,

there is a thermometer named t1 that is located in the room r1

A simple sentence that establishes the fact that a thermometer exists in a given room.

the thermometer t1 can measure the environment variable temperature

Each agent in the system builds its own model of the world that can be used to define concepts such thermometer, temperature, room and so on. As the model is itself defined in CE, the agents build their models through conversing in CE.

there is a radiator valve v1 that is located in the room r1
the radiator valve v1 can control the environment variable temperature

It is also able to using reasoning to determine new facts.

the room r1 has the environment variable temperature that can be measured and that can be controlled

As part of some research work with Cardiff University, we’ve been looking at how CE can be extended to a conversational style of interaction.

These range from exchanging facts between devices – the tell

the environment variable temperature in room r1 has value "21"

Being able to ask question – ask-tell

for which D1 is it true that
      ( the device D1 is located in room V1 ) and
      ( the device D1 can measure the environment variable temperature ) and
      ( the value V1 == "r1")

Expanding on and explaining why certain facts are believed to be true:

the room r1 has the environment variable temperature that can be measured and that can be controlled
    because
the thermometer named t1 is located in the room r1 and can measure the environment variable temperature
    and
the radiator valve v1 is located in the room r1 and can control the environment variable temperature

The fact that the devices communicate in CE means the user can passively observe the interactions. But whilst CE is human readable, it isn’t necessarily human writeable. So some of the research is also looking at how to bridge from NL to CE using a confirm interaction:

NL: The thermometer in the living room has moved to the dining room
CE: the thermometer t1 is located in the room r2

Whilst the current research work is focused on scenarios for civic agencies – for example managing information exchange in a policing context, I’m interested in applying this work to the IoT domain.

With these pieces, you can begin to see how you could have an interaction like this:

    User: I will be late home tonight.
    House: the house will have a state of occupied at 1900
    User: confirmed
    House: the room r1 has a temperature with minimum allowable value 20 after time 1900
           the roomba, vc1, has a clean cycle scheduled for time 1800

Of course this is still quite dry and formal. It would be much more human, more engaging, if the devices came with their own genuine people personality. Or at least, the appearance of one.

    User: I will be late home tonight.
    House: Sorry to hear that, shall I tell everyone to expect you back by 7?
    User: yes please    
    Thermometer: I'll make sure its warm when you get home
    Roomba: *grumble*

I always picture the Roomba as being a morose, reticent creature who really hates its own existence. We have one in the kitchen next to our lab at work, set to clean at 2am. If we leave the door to the lab open, it heads in and, without fail, maroons itself on a set of bar stools we have with a sloped base. Some might call that a fault in its programming, much like Marvin, but I like to think its just trying to find a way to end it all.

This is all some way from having a fully interactive chat room for your devices. But the building blocks are there and I’ll be exploring them some more.

A Conversational Internet of Things – ThingMonk talk

Earlier this year, Tom Coates wrote a blog post about his session at this year’s O’Reilly Foo Camp. Over tea with colleagues, we talked about some of the ideas from the post and how some of our research work might be interesting when applied to them.

One thing led to another and I found myself talking about it at ThingMonk this year. What follows is a slightly expanded version of my talk.


ciot-0

ciot-2

Humanising Things

We have a traditional of putting human faces on things. Whether it’s literally seeing faces on the Things in our everyday lives, such as the drunk octopus spoiling for a fight, or possibly the most scary drain pipe ever.

Equally, we have a tendency to put a human persona onto things. The advent of Twitter brought an onslaught of Things coming online. It seemingly isn’t possible for me to do a talk without at least a fleeting mention of Andy Standford-Clark’s twittering ferries; where regular updates are provided for where each ferry is.

ciot-5

One of the earliest Things on Twitter was Tower Bridge. Tom Armitage, who was working near to the bridge at the time, wrote some code that grabbed the schedule for the bridge opening and closing times, and created the account to relay that information.

ciot-6

One key difference between the ferries and the bridge is that the ferries are just relaying information, a timestamp and a position, whereas the bridge is speaking to us in the first-person. This small difference immediately begins to bring a more human side to the account.
But ultimately, they are simple accounts that relay their state with whomever is following them.

This sort of thing seems to have caught on particularly with the various space agencies. We no longer appear able to send a robot to Mars, or land a probe on a comet without an accompanying twitter account bringing character to the events.

ciot-7

There’s always a sense of excitement when these inanimate objects start to have a conversation with one another. The conversations between the philae lander and its orbiter were particularly touching as they waved goodbye to one another. Imagine, the lander, which was launched into space years before Twitter existed, chose to use its last few milliamps of power to send a final goodbye.

ciot-8

But of course as soon as you peek behind the curtain, you see someone running Tweetdeck, logged in and typing away. I was watching the live stream as the ESA team were nervously awaiting to hear from philae. And I noticed the guy in the foreground, not focused on the instrumentation as his colleagues were, but rather concentrating on his phone. Was he the main behind the curtain, preparing Philae’s first tweet from the surface? Probably not, but for the purposes of this talk, let’s pretend he was.

ciot-9

The idea of giving Things a human personality isn’t a new idea. There is a wealth of rigorous scientific research in this area.

One esteemed academic, Douglas Adams, tells us about the work done by the The Sirius Cybernetics Corporation, who invented a concept called Genuine People Personalities (“GPP”) which imbue their products with intelligence and emotion.

He writes:

Thus not only do doors open and close, but they thank their users for using them, or sigh with the satisfaction of a job well done. Other examples of Sirius Cybernetics Corporation’s record with sentient technology include an armada of neurotic elevators, hyperactive ships’ computers and perhaps most famously of all, Marvin the Paranoid Android. Marvin is a prototype for the GPP feature, and his depression and “terrible pain in all the diodes down his left side” are due to unresolved flaws in his programming.

In a related field, we have the Talkie Toaster created by Crapola, Inc and seen aboard Red Dwarf. The novelty kitchen appliance was, on top of being defective, only designed to provide light conversation at breakfast time, and as such it was totally single-minded and tried to steer every conversation to the subject of toast.

ciot-13

Seam[less|ful]ness

In this era of the Internet of Things, we talk about a future where our homes and workplaces are full of connected devices, sharing their data, making decisions, collaborating to make our lives ‘better’.

Whilst there are people who celebrate this invisible ubiquity and utility of computing, the reality is going to much more messy.

Mark Weiser, Chief Scientist at Xerox PARC, coined the term “ubiquitous computing” in 1988.

Ubiquitous computing names the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives.

Discussion of Ubiquitous Computing often celebrated the idea of seamless experiences between the various devices occupying our lives. But in reality, Mark Weiser advocated for the opposite; that seamlessness was undesirable and self-defeating attribute of such a system.

He preferred a vision of “Seamfulness, with beautiful seams”

ciot-15

The desire to present a single view of the system, with no joins, is an unrealistic aspiration in the face of the cold realities of wifi connectivity, battery life, system reliability and whether the Cloud is currently turned on.

Presenting a user with a completely monolithic system gives them no opportunity to connect with and begin to understand the constituent parts. That is not it say this information is needed to all users all of the time. But there is clearly utility to some users some of the time.

When you come home from work and the house is cold, what went wrong? Did the thermostat in the living room break and decide it was the right temperature already? Did the message from the working thermostat fail to get to the boiler? Is the boiler broken? Did you forgot to cancel the entry in your calendar saying you’d be late home that day?

Without some appreciation of the moving parts in a system, how can a user feel any ownership or empowerment when something goes wrong with it. Or worse yet, how can they avoid feeling anything other than intimidated by this monolithic system that simply says “I’m Sorry Dave, I’m afraid I can’t do that”.

Tom Armitage wrote up his talk from Web Directions South and published it earlier this week, just as I was writing this talk. He covers a lot of what I’m talking about here so much more eloquently than I am – go read it. One piece his post pointed me at that I hadn’t seen was Techcrunch’s recent review of August’s Smart Lock.

ciot-16

Tom picked out some choice quotes from the review which I’ll share here:

“…much of the utility of the lock was negated by the fact that I have roommates and not all of them were willing or able to download the app to test it out with me […] My dream of using Auto-Unlock was stymied basically because my roommates are luddites.”

“Every now and then it didn’t recognize my phone as I approached the door.”

“There was also one late night when a stranger opened the door and walked into the house when August should have auto-locked the door.”

This is the reason for having beautiful seams; seams help you understand the edges of a devices sphere of interaction, but should not be so big to trip you up. Many similar issues exists with IP connected light bulbs. When I need to remember which app to launch on my phone depending on which room I’m walking into, and which bulbs happen to be in there, the seams have gotten too big.

In a recent blog post, Tom Coates wrote about the idea of a chatroom for the house – go read it.

Much like a conference might have a chatroom, so might a home. And it might be a space that you could duck into as you pleased to see what was going on. By turning the responses into human language you could make the actions of the objects less inscrutable and difficult to understand.

ciot-17

This echoes back to the world of Twitter accounts for Things. But rather than them being one-sided conversations presenting raw data in a more consumable form, or Wizard-of-Oz style man-behind-the-curtain accounts, a chatroom is a space where the conversation can flow both ways; both between the owner and their devices, but also between the devices themselves.

What might it take to turn such a chatroom into a reality?

ciot-18

Getting Things Talking

Getting Things connected is no easy task.

We’re still in the early days of the protocol wars.

Whilst I have to declare allegiance to the now international OASIS standard MQTT, I’m certainly not someone who thinks one protocol will rule them all. It pains me whenever I see people make those sorts of claims. But that’s a talk for a different day.

Whatever your protocol of choice, there are an emerging core set that seem to be the more commonly talked about. Each with its strengths and weaknesses. Each with its backers and detractors.

ciot-19

What (mostly) everyone agrees on is the need for more than just efficient protocols for the Things to communicate by. A protocol is like a telephone line. It’s great that you and I have agreed on the same standards so when I dial this number, you answer. But what do we say to each other once we’re connected? A common protocol does not mean I understand what you’re trying to say to me.

And thus began the IoT meta-model war.

There certainly a lot of interesting work being done in this area.

For example, HyperCat, a consortium of companies coming out of a Technology Strategy Board funded Demonstrator project in the last year or so.

ciot-21

HyperCat is an open, lightweight JSON-based hypermedia catalogue format for exposing collections of URIs. Each HyperCat catalogue may expose any number of URIs, each with any number of RDF-like triple statements about it. HyperCat is simple to work with and allows developers to publish linked-data descriptions of resources.

URIs are great. The web is made of them and they are well understood. At least, they are well understood by machines. What we’re lacking is the human view of this world. How can this well-formed, neatly indented JSON be meaningful or helpful to the user who is trying to understand what is happening.

This is by no means a criticism of HyperCat, or any of the other efforts to create models of the IoT. They are simply trying to solve a different set of problems to the ones I’m talking about today.

ciot-23

Talking to Computers

We live in an age where the talking to computers is becoming less the reserve of science fiction.

Siri, OK Google, Cortana all exist as ways to interact with the devices in your pocket. My four year old son walks up to me when I have my phone out and says: “OK Google, show me a picture of the Octonauts” and takes over my phone without even having to touch it. To him, as to me, voice control is still a novelty. But I wonder what his 6 month old sister will find to be the intuitive way of interacting with devices in a few years time.

The challenge of Natural Language Parsing, NLP, is one of the big challenges in Computer Science. Correctly identifying the words being spoken is relatively well solved. But understanding what those words mean, what intent they try to convey, is still a hard thing to do.

To answer the question “Which bat is your favourite?” without any context is hard to do. Are we talking to a sportsman with their proud collection of cricket bats? Is it the zoo keeper with their colony of winged animals. Or perhaps a comic book fan being asked to chose between George Clooney and Val Kilmer.

Context is also key when you want to hold a conversation. The English language is riddled with ambiguity. Our brains are constantly filling in gaps, making theories and assertions over what the other person is saying. The spoken word also presents its own challenges over the written word.

ciot-28

“Hu was the premiere of China until 2012″

When said aloud, you don’t know if I’ve asked you a question or stated a fact. When written down, it is much clearer.

ciot-29

In their emerging technology report for 2014, Gartner put the Internet of Things at the peak of inflated expectation. But if you look closely at the curve, up at the peak, right next to IoT, is NLP Question Answering. If this was a different talk, I’d tell you all about how IBM Watson is solving those challenges. But this isn’t that talk.

ciot-30

A Conversational Internet of Things

To side step a lot of the challenges of NLP, one area of research we’re involved with is that of Controlled Natural Language and in particular, Controlled English.

CE is designed to be readable by a native English speaker whilst representing information in a structured and unambiguous form. It is structured by following a simple but fully defined syntax, which may be parsed by a computer system.

It is unambiguous by using only words that are defined as part of a conceptual model.

CE serves as a language that is both understandable by human and computer system – which allows them to communicate.

For example,

there is a thermometer named t1 that is located in the room r1

A simple sentence that establishes the fact that a thermometer exists in a given room.

the thermometer t1 can measure the environment variable temperature

Each agent in the system builds its own model of the world that can be used to define concepts such thermometer, temperature, room and so on. As the model is itself defined in CE, the agents build their models through conversing in CE.

there is a radiator valve v1 that is located in the room r1
the radiator valve v1 can control the environment variable temperature

It is also able to using reasoning to determine new facts.

the room r1 has the environment variable temperature that can be measured and that can be controlled

As part of some research work with Cardiff University, we’ve been looking at how CE can be extended to a conversational style of interaction.

These range from exchanging facts between devices – the tell

the environment variable temperature in room r1 has value "21"

Being able to ask question – ask-tell

for which D1 is it true that
      ( the device D1 is located in room V1 ) and
      ( the device D1 can measure the environment variable temperature ) and
      ( the value V1 == "r1")

Expanding on and explaining why certain facts are believed to be true:

the room r1 has the environment variable temperature that can be measured and that can be controlled
    because
the thermometer named t1 is located in the room r1 and can measure the environment variable temperature
    and
the radiator valve v1 is located in the room r1 and can control the environment variable temperature

The fact that the devices communicate in CE means the user can passively observe the interactions. But whilst CE is human readable, it isn’t necessarily human writeable. So some of the research is also looking at how to bridge from NL to CE using a confirm interaction:

NL: The thermometer in the living room has moved to the dining room
CE: the thermometer t1 is located in the room r2

Whilst the current research work is focused on scenarios for civic agencies – for example managing information exchange in a policing context, I’m interested in applying this work to the IoT domain.

With these pieces, you can begin to see how you could have an interaction like this:

    User: I will be late home tonight.
    House: the house will have a state of occupied at 1900
    User: confirmed
    House: the room r1 has a temperature with minimum allowable value 20 after time 1900
           the roomba, vc1, has a clean cycle scheduled for time 1800

Of course this is still quite dry and formal. It would be much more human, more engaging, if the devices came with their own genuine people personality. Or at least, the appearance of one.

    User: I will be late home tonight.
    House: Sorry to hear that, shall I tell everyone to expect you back by 7?
    User: yes please    
    Thermometer: I'll make sure its warm when you get home
    Roomba: *grumble*

I always picture the Roomba as being a morose, reticent creature who really hates its own existence. We have one in the kitchen next to our lab at work, set to clean at 2am. If we leave the door to the lab open, it heads in and, without fail, maroons itself on a set of bar stools we have with a sloped base. Some might call that a fault in its programming, much like Marvin, but I like to think its just trying to find a way to end it all.

This is all some way from having a fully interactive chat room for your devices. But the building blocks are there and I’ll be exploring them some more.

Tackling Cancer with Machine Learning

For a recent Hack Day at work I spent some time working with one of my colleagues, Adrian Lee, on a little side project to see if we could detect cancer cells in a biopsy image.  We've only spent a couple of days on this so far but already the results are looking very promising with each of us working on a distinctly different part of the overall idea.

We held an open day in our department at work last month and I gave a lightening talk on the subject which you can see on YouTube:


There were a whole load of other talks given on the day that can be seen in the summary blog post over on the ETS (Emerging Technology Services) site.



Went to Designcamp, got the T-Shirt

Last month, I was fortunate enough to fly off to Austin with a group of colleagues for a week long IBM Design Thinking camp. It was an opportunity to get away from the day job, with laptops all-but banned, and have a deep-dive into what IBM Design is about and how it can be applied.

As a relatively new effort within the company, IBM Design sets out to bring a focus back to where it should be; the human-experience of our products and services. This isn’t just about making pretty user interfaces; it is the entire experience of our products.

As an engineer, the temptation is always there to create shiny new features. But no matter how shiny it is, if it isn’t what a user needs, then it’s a waste of effort. The focus has to be on what the user wants to be able to do. This is something I’ve always tried to do with Node-RED; we often get suggestions for features that, once you start picking at them, are really solutions looking for a problem. Once you work back and identify the problem, we’re often able to identify alternative solutions that are even better.

P1070048

It’s often just a matter of asking the right question; At Designcamp, the very first exercise we were asked to do was to draw a new type of vase. Everyone drew something that looked vaguely vase-like. Then (spoilers…) we were asked to draw a better way to display flowers. At this point we got lots of decidedly un-vase-like ideas that were much more imaginative. It’s the difference between asking for a feature and asking for an idea. The former presupposes a lot about the nature of the answer, the latter is focused on not just the what, but also the why.

This relentless focus on the user isn’t a new idea. GDS, who are doing incredible things with government services, have it as their very first Design Principle. But it is refreshing to see this focus being brought to bear within a transformation of how the entire company operates.

Oh, and of course being in Austin, we got to screen print our own IBM Designcamp T-Shirts to commemorate the visit.

Go to Designcamp, screen print your own t-shirt. Obvs.

Lots more photos from the week over on flickr.

Hursley 3D Printing Expo

D’oh, looks like I missed a swarm of 3d printers in Hursley recently! I wonder if anyone has printed a model of the house/site yet.

I’m still looking for even a vaguely plausible excuse to splash out on a 3d printer, but printing models or new 3d printers still isn’t quite enough to justify the money (or space these days)!


Kids should learn to code

Does a five-year-old need to learn how to code?

A couple of weeks ago I was interviewed by the BBC. In a fairly long phone call, I either rambled inanely or provided detailed and nuanced answers in context. That depends on your point of view.

Either way, obviously not a lot of it could make it into their story, as they really only needed a few quotes. So I thought I’d put more of what I said here.

The background for the story was the changes to the UK school curriculum which means that all kids are being taught to code. And the basic premise for the piece was that as we’re “entering an era when computers are actually beginning to teach themselves” that this is unnecessary and that coding itself is becoming an outdated skill.

This is a summary of what I tried to say…

Learning to “code”

It’s useful to start with some context. When we talk about teaching kids to “code” we don’t just mean teaching them how to write lines of code – it’s broader than that. Some criticisms of this initiative seem to be arguing against five-year olds needing to learn where to put semi-colons, which is missing the point.

From what I’ve seen, it’s an umbrella term that covers a range of activities such as:

Logical thinking and problem solving

Teaching kids how to understand a description of a problem, identify a solution, and describe that solution by breaking it down into a series of steps.

As kids get older this can be framed as how to write an algorithm. But it’s something that can be started even at Faith’s age (6) and without needing to touch a keyboard. That’s not new – how many developers have had to answer the interview question “describe how to make a cup of tea”?

You don’t need to learn programming language syntax to start getting your head around this, and I would argue it’s a vital skill to develop in life, even if you don’t become a coder.

Technological creativity

We need to do more than teach children how to use the tools that they have today. We need to encourage an ethos from an early age that we don’t have to be passive users of technology.

It’s about teaching kids how to think of and how to approach technology. They don’t have to think of it as a black box that must be used as-is, but as something that they can remix and tweak and modify and change and create. It’s about an attitude of looking at technology as something that they can make do what they want to do, as opposed to use the way someone tells them they should.

This is what I love about running my Code Club. Instead of kids playing a random Flash game they find online, they can make a game themselves, the way they want it to be. If they want it to be faster, slower, bigger, smaller, a different colour, move differently: they are in control. It’s not fixed, they can make it do and behave the way they want it to. And if they realise that they can do that with technology, it’s a real light-bulb moment.

We need kids to have this mindset so they will grow up able to imagine the next wave of innovations. Saying that we don’t need this because we can delegate it to the computers we have today really feels to me to be missing the point. Cognitive computing holds exciting promise and potential but it does not mean “we won’t need to be creative any more, the computers will do that for us, too”.

Coding becoming “outdated”

Leaving aside this bigger picture, is coding itself a useful skill to learn. Is coding going to become outdated?

I don’t think so.

Part of this argument seemed to be “what is the point of teaching kids <insert-name-of-programming-language-here> because by the time they grow up it will be obsolete?”

Programming languages stick around longer than people think – there are people still making a living writing C and maintaining COBOL. (We’re normally after good Prolog people, too!)

But more importantly, a lot of what you learn in one language is transferable. Every time I’ve started working in a new programming language, I’ve built on the basic concepts I already know from others. Maybe we’ll teach children a programming language that isn’t the most widely used language when they’re older. But that doesn’t mean learning the underlying ideas will have been a waste of time.

The argument also seemed to be that not just any particular language, but coding in general will become obsolete. I’m not convinced by this.

What we mean by coding may be different in twenty years to what we mean today. In fact it probably will be. Coding will evolve. It always has, and I’m sure it will continue to.

Even just looking at my personal coding history, you can see that evolution. Writing in assembler (where I was moving data in and out of registers) was different to writing in C. And writing in C (where it wasn’t just about what I wanted it to do functionally, but also doing my own memory management) was different to my coding today in Java.

A big difference is in the level of abstraction. They all involved describing to the computer something that I wanted it to do. But the level of abstraction I’m able to use to describe it has changed.

I’m sure this is a trend that will continue. New programming languages will get higher and higher level. Future programming languages will give us ways to describe what we want with higher levels of abstraction. And maybe that will look closer to natural language than what we have today (well-written Java is already closer to being readable by a lay-person than assembler). Maybe it will be something like a Controlled English language that feels more like describing what you want to another person.

But that won’t mean that coding has become obsolete, just that it will have evolved as it always has.

The need for people who can understand a problem, and describe to a computer how to solve it, will remain – whatever language they use and whether that language looks like “code” as we understand it today.


Unpacking binary data from MQTT in Javascript

While doing trawl of Stackoverflow for questions I might be able to help out with I came across this interesting looking question:

Receive binary with paho mqttws31.js

The question was how to unpack binary MQTT payloads into double precision floating point numbers in javascript when using the Paho MQTT over WebSockets client.

Normally I would just send floating point numbers as strings and parse them on the receiving end, but sending them as raw binary means much smaller messages, so I thought I’d see if I could help to find a solution.

A little bit of Googling turned up this link to the Javascript typed arrays which looked like it probably be in the right direction. At that point I got called away to look at something else so I stuck a quick answer in with a link and the following code snippet.

function onMessageArrived(message) {
  var payload = message.payloadByte()
  var doubleView = new Float64Array(payload);
  var number = doubleView[0];
  console.log(number);
}

Towards the end of the day I managed to have a look back and there was a comment from the original poster saying that the sample didn’t work. At that point I decided to write a simple little testcase.

First up quick little Java app to generate the messages.

import java.nio.ByteBuffer;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;

public class MessageSource {

  public static void main(String[] args) {
    try {
      MqttClient client = new MqttClient("tcp://localhost:1883", "doubleSource");
      client.connect();

      MqttMessage message = new MqttMessage();
      ByteBuffer buffer = ByteBuffer.allocate(8);
      buffer.putDouble(Math.PI);
      System.err.println(buffer.position() + "/" + buffer.limit());
      message.setPayload(buffer.array());
      client.publish("doubles", message);
      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }
      client.disconnect();
    } catch (MqttException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
}

It turns out that using typed arrays is a little more complicated and requires a bit of work to populate the data structures properly. First you need to create an ArrayBuffer of the right size, then wrap it in a Uint8Array in order to populate it, before changing to the Float64Array. After a little bit of playing around I got to this:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[i] = payload[i];
  }
  var doubleView = new Float64Array(uint.buffer);
  var number = doubleView[0];
  console.log("onMessageArrived:"+number);
};

But this was returning 3.207375630676366e-192 instead of Pi. A little more head scratching and the idea of checking the byte order kicked in:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[(length-1)-i] = payload[i];
  }
  var doubleView = new Float64Array(uint.buffer);
  var number = doubleView[0];
  console.log("onMessageArrived:"+number);
};

This now gave an answer of 3.141592653589793 which looked a lot better. I still think there may be a cleaner way to do with using a DataView object, but that’s enough for a Friday night.

EDIT:

Got up this morning having slept on it and came up with this:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[i] = payload[i];
  }
  var dataView = new DataView(uint.buffer);
  for (var i=0; i<length/8; i++) {
      console.log(dataView.getFloat64((i*8), false));
  }
};

This better fits the original question in that it will decode an arbitrary length array of doubles and since we know that Java is big endian, we can set the little endian flag to false to get the right conversion without having to re-order the array as we copy it into the buffer (which I’m pretty sure wouldn’t have worked for more than one value).

Talking about IBM Watson (again)

As I mentioned in May, I was lucky to be able to go to Thinking Digital this year and talk about what we’re doing with Watson.

I’ve just noticed that they’ve made a video of my talk available. I haven’t dared watch it (does anyone like watching videos of themselves?), but I figured I should share it anyway!


Thinking Digital 2014

This week I went up to Newcastle for Thinking Digital.

It was the seventh Thinking Digital, but my first.

I’d seen a bunch of references to it being the UK’s answer to TED, the tickets aren’t cheap, videos from previous years look slick and professional, it’s held in The Sage which is a hugely impressive venue, they manage to get a great line-up of speakers, and the logistics in the run-up to the event were more organised than any event I’ve been to before.

So… I was expecting a cool and geeky, if faceless, serious, formal, and intimidating event.

I’d read it completely wrong. It’s absolutely a professionally run event. And there was no shortage of cool geekiness. But, more than that, the organizer, Herb Kim, has created a real sense of community in it. There’s a feeling of almost familial warmth amongst attendees who come year after year after year.

And they do it without being too cliquey. Everyone I spoke to was very friendly and welcoming, which made the few days a lot easier for an introvert like me. A few days being surrounded by and trying to talk to and socialise with several hundred smart brilliant people is the kind of thing I normally find hugely draining and more than a little daunting. But the crowd at TDC make it easier than most.

They value their time there, too. More than one person told me they’d paid for their own ticket and expenses to attend. I’m used to corporate-run conferences where everyone is paid for by their employer, or barcamps where people moan about being asked for a five pound deposit, so this surprised me.

The talks made for a fascinating and thought-provoking couple of days. I can’t do them justice here (when videos of the talks are available I’ll embed/link them here instead) but I want to give an idea of what the programme was like.

Jeni TennisonOpen Data Institute
Talked about the potential impact of open data on society, giving examples of how open data could be used to inform and widen access to debate.

Maik MaurerSpritz
Demonstrated their speed-reading technology – streaming one word at a time in a fixed place, for fast reading on mobile and wearable devices.

Gerard GrechTech City
Talked about the role of Tech City as a feedback loop between Government and the tech community.

Meri WilliamsChromeRose
Talked about the lessons that people managers could learn from artificial intelligence in how to inspire, motivate, and enable geeks to achieve great things.

Aral Balkanindie Phone
Gave an impassioned and stirring talk entitled “Free is a Lie” about the conflict between advertising-led business models, and user’s privacy and other interests.

David Griffithsfoam
Talked about using his background in the video game industry to combine crowd-sourcing and gaming to perform impressive citizen science projects.

Chi OnwurahMP for Newcastle Central
Talked about the parallels between technology and politics as driving forces for change, and the aims of the current Digital Government Review.

Mariana MazzucatoUniversity of Sussex
Argued that the image of the private sector as entrepreneurial and public sector as meddling and restrictive is an unhelpful myth and made the case for a bolder, entrepreneurial state.

Erin McKeanWordnik
Talked about the limitations of search as a model for accessing data and the need for discovery engines to find what you don’t know you want.

Blaise Aguera y ArcasGoogle
Described the history of machine intelligence and his predictions about what the future of machine intelligence might look like.

Carl LedbetterMicrosoft
Outlined the history and evolution of digital entertainment, and described the process that went into the design of the XBox One.

Jennifer GardyBC Centre for Disease Control
Described our progress in increasing our understanding of the human genome, and where it’s complexity lies.

Peter Gregson – Cellist
Gave a representation of the genome work that Jennifer had described. Instead of a data visualisation, it was a sonification. Using a cello.

Sean CarassoFalling Whistles
Told an inspiring story of how he came to learn about the terrible things happening in Congo, and how he went about trying to bring peace.

Conrad BodmanThe Barbican
Argued for recognition of the impact of digital tech on the arts, and described his projects to exhibit and showcase video games, animation, and digital effects.

Mark DearnleyHMRC
Described the challenges and need for technology in what HMRC do, and their digital ambition for the future.

Xavier De KestellerFoster + Partners
Talked about an amazing project to build a base on the moon, using autonomous robots with 3D printing heads to print a building out of moon dust.

Susan MulcahyImperial College London
Gave an energetic performance to describe the role of the red blood cell, and the science behind understanding brain injury.

Carlos UlloaHelloEnjoy
Showed what was possible using WebGL, bringing native 3D gaming to the browser without the need for plugins.

Jonathan O’HalloranQuantuMD
Described his work to create a mobile genetic-testing device, and the potential that real-time epidemiology from a mobile device could bring.

Blaise Aguera y ArcasGoogle
Talked about changes needed in society when more jobs are replaced by technology, and his observations about changes in gender dynamics.

Steve MouldBBC
Gave an entertaining talk about how he discovered, and tried to understand the science behind, the bead chain fountain.

Tom ScottUs Vs Th3m
Ended the conference with a fantastic performance showing what the impact of technology might be like in 2030.

Dale LaneIBM
And I did a Watson talk. I really didn’t want it to seem like a sales pitch, so I tried to put it in a bigger context of being a step forwards in changing how we use computers. I talked about why I work on Watson, what motivates and inspires me about it, and why I think what we’re doing is difficult but hopefully valuable. And I walked through a short demo to explain the value I see in where we are even now. Annoying technical issues (Keynote + clicker + multiple screens = fail) aside, it went okay. It was a lot to try and fit into 20 minutes, so I talked fast. :-)

Overall…

It was a fantastic event, and one I’d wholeheartedly recommend.

If you can get to a future Thinking Digital, you absolutely should.

It’s one of the most thought-provoking and interesting couple of days I’ve had in a long time.

.

Full-diclosure: As a speaker, I didn’t have to pay for a ticket to attend this event. My travel and accommodation costs were paid for by IBM.


Text analytics in BlueMix using UIMA

In this post, I want to explain how to create a text analytics application in BlueMix using UIMA, and share sample code to show how to get started.

First, some background if you’re unfamiliar with the jargon.

What is UIMA?

UIMA (Unstructured Information Management Architecture) is an Apache framework for building analytics applications for unstructured information and the OASIS standard for content analytics.

I’ve written about it before, having used it on a few projects when I was in ETS, and on other side projects since such as building a conversational interface to web pages.

It’s perhaps better known for providing the architecture for the question answering system IBM Watson.

What is BlueMix?

BlueMix is IBM’s new Platform-as-a-Service (PaaS) offering, built on top of Cloud Foundry to provide a cloud development platform.

It’s in open beta at the moment, so you can sign up and have a play.

I’ve never used BlueMix before, or Cloud Foundry for that matter, so this was a chance for me to write my first app for it.

A UIMA “Hello World” for BlueMix

I’ve written a small sample to show how UIMA and BlueMix can work together. It provides a REST API that you can submit text to, and get back a JSON response with some attributes found in the text (long words, capitalised words, and strings that look like email addresses).

The “analytics” that the app is doing is trivial at best, but this is just a Hello World. For now my aim isn’t to produce a useful analytics solution, but to walk through the configuration needed to define a UIMA analytics pipeline, wrap it in a REST API using Wink, and deploy it as a BlueMix application.

When I get a chance, I’ll write a follow-up post on making something more useful.

You can try out the sample on BlueMix as it’s deployed to bluemix.net

The source is on GitHub at github.com/dalelane/bluemixuima.

In the rest of this post, I’ll walk through some of the implementation details.

Runtimes and services

Creating an application in BlueMix is already well documented so I won’t reiterate those steps, other than to say that as Apache UIMA is a Java SDK and framework, I use the Liberty for Java runtime.

I’m not using any of the services in this simple sample.

Manifest

The app is bundled up in a war file, which is what we deploy. This is specified in manifest.yml.

Building

The war file is built by an ant task which has to include the UIMA jar in the classpath, and copy my UIMA descriptor XML files into the war.

I’m developing in eclipse, so I set up an ant builder to run the build, and configured the project to do it automatically.

I’m deploying from eclipse, too, using the Cloud Foundry plugins for eclipse.

XML descriptors

The type system is defined in an XML descriptor file and specifies the different annotations that can be created by this pipeline, and the attributes that they have.

Running JCasGen in eclipse on that descriptor generates Java classes representing those types.

The pipeline is also defined in XML descriptors: one overall aggregate descriptor which imports three primitive descriptors for each of the three annotators in my sample pipeline : one to find email addresses, one to find capitalised words and one to find long words.

Note that the imports in the aggregate descriptor need to be relative so that they keep working once you deploy to BlueMix.

These XML descriptor files are all added to the war file by being included in the build.xml with a fileset include.

Annotators

Each of the primitive descriptor files specifies the fully qualified class name for the Java implementation of the annotator.

There are three annotators in this sample. (XML files with names starting “primitiveAeDescriptor”).

Each one is implemented by a Java class that extends JCasAnnotator_ImplBase.

Each uses a regular expression to find things to annotate in the text. This isn’t intended to be an indication that this is how things should be done, just that it makes for a simple and stateless demonstration without any additional dependencies.

The simplest is the regex used to find capitalised words in WordCaseAnnotator and the most complex is the ridiculously painful one used to find email addresses in EmailAnnotator.

Note that the regexes are prepared in the annotator initializer, and reused for each new CAS to process, to improve performance.

UIMA pipeline

The UIMA pipeline is defined in a single Java class.

It finds the XML descriptor for the pipeline by looking in the location where BlueMix will unpack the war.

It creates a CAS pool to make it easier to handle multiple concurrent requests, and avoid the overhead of creating a CAS for every request.

Once the pipeline is initialised, it is ready to handle incoming analysis requests.

Once the CAS has passed through the pipeline, the annotations are immediately copied out of the CAS into a POJO, so that the CAS can be returned to the pool.

REST API

The war file deployed to BlueMix contains a web.xml which specifies the servlet that implements the REST API.

I’m using Wink to implement the API. The servlet definition in the web.xml specifies where to find the list of API endpoints and the URL where the API should be.

The list of API endpoints is a list of classes that Wink uses. There is only one API endpoint, so only one class listed.

The API implementation is a very thin wrapper around the Pipeline class.

Everything is defined using annotations, and Wink handles turning the response into a JSON payload.

That’s it

I think that’s pretty much it.

I’ve added a simple front-end webpage, with a script to submit API requests for people who don’t want to do it with something like curl.

It’s live at uimahelloworld.mybluemix.net.

Like I said, it’s very simple. The Java itself isn’t particularly complex. My reason for sharing it was to provide a boilerplate config for defining a UIMA analytics pipeline, wrapping it in a REST API, and deploying it to BlueMix.

Once you’ve got that working, you can do text analytics in BlueMix as complex as whatever you can dream up for your annotators.

When I get time, I’ll write a follow-up post sharing what that could look like.