Saturday, June 27, 2009

Homemade Python Container Object


I wrote what I think is a nifty piece of code, and thought I'd share it, and get some feedback. The problem I was struggling with was that I wanted a flexible container object that other developers could use for storage and retrieval of objects, without pre-defining the structure.

In other words, I wanted it to be possible to do something like this:
if o.chatperson.hometown.country == "Mexico":
o.chatperson.native_language = "Spanish"
else:
o.chatperson.native_language = "English"


The code above examines the country of the hometown of the person that Amy Iris is chatting with, and if it's Mexico, then set their native language to Spanish, otherwise it sets it to English.

So far, not hard. The catch was that I wanted to be able to create and query the objects and properties on the fly. Using typical Python classes, the "if" statement above would raise an error, unless you have o defined, and o has a property of chatperson, and o.chatperson has a property of hometown, and o.chatperson.hometown has a property of country. I'd rather it not raise an error, but instead, create the hierarchy of sub-properties as needed to complete the request.

My goal was to create a general purpose container object that would create sub-containers on the fly, so that such a query wouldn't fail, and more importantly, it would allow me to set "deeply nested" properties, without having to pre-define the entire hierarchical structure.

So if I define this class properly, and set up an instance o, I should be able to execute the line:
o.chatperson.native_language = "English"

without having to define o.chatperson in advance.

I want this statement to automatically add a property called chatperson if one doesn't already exist, and make it a container to hold the native_language property.

By setting up a container class (I called it "Box"), and having "o" an instance of that class, I should be able to execute that line of code. Upon execution, "o" will create the container "chatperson", which will allow you to set the property called native_language. All auto-magically.

I called my class a "Box" class (looking for a shorter name than Container). The other features that I wanted were:

  • properties and sub-properties would automatically be set up (as instances of Box), if they were queried or needed for an assignment.
  • I wanted to track all changes to any instance of the class at the highest level. So setting o.chatperson.native_language to "English" will also log that change into a list called o._changelog.
  • I wanted to be able to set and get properties by either the dotted notation (in an unquoted object name in my source code), or by string. So the property that I mentioned above could be accessed in a number of ways:

o.chatperson.hometown.country
o.chatperson.hometown.get_dotted("country")
o.chatperson.get_dotted("hometown.country")
o.get_dotted("chatperson.hometown.country")
o.chatperson.get_dotted("hometown").country
o.get_dotted("chatperson.hometown").country
o.get_dotted("chatperson").hometown.country
o.get_dotted("chatperson").hometown.get_dotted("country")


Likewise, o.set_dotted could be used to "deep set" a property (and all necessary parent container properties would automatically be built as needed). And o.del_dotted could be used to delete using a string to navigate down the object heirarchy.

So here's what I came up with. Feel free to use it or critique it:


class Box:
"""
Containers for Properties. This class will allow you to create instances
that contain properties and sub-properties.

It has an auto-create feature, so that if you use dotted notation and
reference properties and sub-properties that do not yet exist,
they are created automatically.

And you can create dotted-notation properties using variable names.

All changes are tracked in a _changelog list.

Examples:


>>> p1=Box(name="Chris",gender="male")
>>> p1
{'gender': 'male', 'name': 'Chris'}
>>> p2=Box(name="Joe",hometown="Cincinnati")
>>> p2
{'hometown': 'Cincinnati', 'name': 'Joe'}
>>> PeopleBox=Box(people=[p1,p2])
>>> PeopleBox
{'people': [{'gender': 'male', 'name': 'Chris'}, {'hometown': 'Cincinnati', 'name': 'Joe'}]}
>>> p2.contact_info.phone.home="513-555-1111" #auto-create contact_info.phone
>>> p2
{'hometown': 'Cincinnati', 'name': 'Joe', 'contact_info': {'phone': {'home': '513-555-1111'}}}
>>> p1.family_members=[p2,] #support list types
>>> p1
{'gender': 'male', 'name': 'Chris', 'family_members': [{'hometown': 'Cincinnati', 'name': 'Joe', 'contact_info': {'phone': {'home': '513-555-1111'}}}]}
>>> p1.family_members[0].contact_info.phone.mobile #access p2 through p2
{}
>>> p2.contact_info.phone.mobile="513-555-2222"
>>> p2
{'hometown': 'Cincinnati', 'name': 'Joe', 'contact_info': {'phone': {'mobile': '513-555-2222', 'home': '513-555-1111'}}}
>>> p1.family_members[0].contact_info.phone.mobile #access p2 through p1
'513-555-2222'
>>> p2.contact_info.phone.mobile='513-555-3333'
>>> p1.family_members[0].contact_info.phone.mobile #changes to p2 affect p1
'513-555-3333'
>>> properties="job.salary"
>>> p1.set_dotted(properties,120000) #can create properties whose dotted names are in an object
>>> p1.job.salary
120000
>>> p2._changelog #can see list of changes
[['setattr', 'contact_info.phone.home'], ['setattr', 'contact_info.phone.mobile'], ['setattr', 'contact_info.phone.mobile']]
>>> p2._all() #can see entire structure
{'name': 'Joe', 'contact_info': {'_top': '<pointer>', 'phone': {'_top': '<pointer>', 'home': '513-555-1111', 'mobile': '513-555-3333', '_name': 'contact_info.phone'}, '_name': 'contact_info'}, '_top': '<pointer>', 'hometown': 'Cincinnati', '_name': '', '_changelog': [['setattr', 'contact_info.phone.home'], ['setattr', 'contact_info.phone.mobile'], ['setattr', 'contact_info.phone.mobile']]}
"""


def __init__(self, **kwargs):
"""
Save any dictionary of keywords passed, into the container
"""
self.__dict__ = kwargs
_top=self.__dict__.setdefault("_top",self)
if id(_top) == id(self):
self.__dict__["_changelog"]=[]
self.__dict__["_name"]=""


def set_dotted(self,name,value):
"""
set a value, by passing a dot-notation property name and value
"""
parts=name.split(".",1)
if self.__dict__.get(parts[0],None).__class__ != Box:
_top = self.__dict__.get("_top",self)
_name = self._get_full_name(parts[0])
self.__dict__[parts[0]]=Box(_top=_top, _name=_name)
if len(parts)==1:
self.__setattr__(name, value)
else:
self.__dict__[parts[0]].set_dotted(parts[1],value)

def del_dotted(self,name):
"""
delete a property, by passing a dot-notation property name
"""
parts=name.split(".",1)
if len(parts)==1:
self.__delattr__(name)
else:
self.__dict__[parts[0]].del_dotted(parts[1])

def get_dotted(self,name):
"""
get a property, by passing a dot-notation property name
"""
parts=name.split(".",1)
if len(parts)==1:
return self.__dict__[name]
else:
return self.__dict__[parts[0]].get_dotted(parts[1])

def __getattr__(self, name):
if name[:2]!="__" or name[-2:]!="__": # skip the magic python ones.
_top = self.__dict__.get("_top",self)
_name = self._get_full_name(name)
self.__dict__[name]=Box(_top=_top, _name=_name) #auto-create
return self.__dict__[name]
else:
raise AttributeError, name

def __setattr__(self, name, value):
self.__dict__[name]=value
_name = self._get_full_name(name)
self.__dict__["_top"].__dict__["_changelog"].append(["setattr",_name,value])

def __delattr__(self, name):
del self.__dict__[name]
_name = self._get_full_name(name)
self.__dict__["_top"].__dict__["_changelog"].append(["delattr",_name])

def _get_full_name(self, name):
"""
determine full dot-notation name of the property
"""
_name = [self.__dict__["_name"]]
if len(_name[0])==0:
_name=[]
_name.append(name)
_name=".".join(_name)
return _name

def __repr__(self):
"""
simple representation, no hidden values
"""
temp={}
for (k,v) in self.__dict__.items():
if k not in ["_name","_changelog","_top"]:
temp[k]=v
return repr(temp)

def _all(self):
"""
return all values including hidden ones
"""
temp={}
for (k,v) in self.__dict__.items():
if k == "_top":
temp[k]="<pointer>"
elif v.__class__ == Box:
temp[k] = v._all()
else:
temp[k] = v
return temp


Sunday, June 21, 2009

Want 330 Worthless Followers?



Four months ago, @lifewithryan told me about accidentally getting unwanted followers simply by tweeting the wrong thing. I set up a test account to see how that might work. I sent out 2 stupid tweets, and got 330 followers over the four months.





This would be a fun contest! See how many followers you can get by sending out just one tweet, over seven days. Rules: you have to set up a new account, and send just one tweet, and then hop over to twittercounter and do a search on your new account so that you are tracked. (to prove it's your new account, you should probably tweet the name of the new account from your usual account.)

I got 330 followers in 4 months with 2 tweets. How many can you get in 7 days with just 1 tweet on a new account?


By the way, in case you are new to this blog, I should tell you that Amy Iris is an extensible bot that you can place on your website, like I have at the top of this blog post.

Talk to my bot! Type into the text box, above.

Friday, June 19, 2009

How Amy Iris knows where Best Buy is

Amy Iris is an extensible bot that you can place on your website, like this:



Feel free to talk to her, by typing into the text box above. If you ask her questions, she'll answer you. The cool thing is that we have designed her for extensibility, so that the internet community can make her smarter. As an example, I built a small code snippet to interface with the Best Buy Remix API so that you can ask her questions about where Best Buy stores are. Here's a picture of a dialog that I had with Amy Iris earlier today:




As you can see, I have asked Amy Iris a couple of different ways to tell me where various Best Buy stores are. This example could be extended for all of the Best Buy API calls (such as product lookups).

Here's the code that's part of the Bot's logic. One little 14-line code snippet, submitted to Amy Iris' brain, and she now is that much smarter.



# example of Best Buy Store Locator
import amyiris, re
from google.appengine.api import urlfetch
if ("best buy" in str(textin)) and ((' in ' in str(textin)) or
('near' in str(textin))):
fields = ['name','distance','address']
url = "http://api.remix.bestbuy.com/v1/stores(area(%s,50))?show="
url += ",".join(fields) + "&apiKey=amysremixkeygoeshere"
r = "The nearest Best Buy store to %s is the %s store, "+
"which is %s miles away, at %s."

vals = [re.search(r'\d'*5,str(textin)).group(0),] #grab the zip code
page = urlfetch.fetch(url%vals[0]).content #look up results based on zipcode
for tag in fields: #parse the xml
vals.append(re.search('<'+tag+'>(.*?)</'+
tag+'>',page, re.DOTALL).group(1))
say(r%tuple(vals),confidence=21 )


A quick code review reveals the tricks (and limitations) of this conversational parser. I scan for the words "best buy", " in ", and "near", and rely on a 5-digit zip code in a regex search (that is, r'\d'*5). And if I find all these, then the snippet will retrieve the information from the Best Buy web site and present it to the user in conversation form.

Imagine - it's now available on the web, on twitter, on your cell phone. And this is just one small look-up. Imagine what happens as people begin contributing similar code snippets over the years! Amy Iris will be brilliant!

Thursday, June 18, 2009

Integrating Amy Iris into your Website

Amy Iris is a community built bot. We have the tools and platform now, so that within a few years, collectively we can build a very smart conversational bot through user contributions. Wikipedia taught us the power of pooling very small contributions of a lot of people. We're trying to use a lot of Wikipedia's principles: Focus on making it free and easy to use and contribute.

Now you can put this widget onto your website!
Test it for me. Ask Amy Iris a question.



As a developer, why would you want to a conversational bot in your project? Smart bots can provide your users and customers with a better experience using your website. Check out Paypal's virtual assistant or any of the bots on chatbot.org, and you can see that bots provide a way to automate the mundane, provide better customer service, and save companies money.

One tool that we've created is a simple Amy Iris widget (above) that you can drop onto your website. In future blog posts, I'll show you how you can make this widget smarter and "application specific". But for now I'll just share the tool.

Here's the widget, and the code necessary to drop it onto your website. Go ahead, try it out. Talk to her! And let me know if this widget is working - I'm still testing it. She should "open a chat window" by expanding her blue box, if you talk to her. Thanks!




<script type="text/javascript">
var oldHistLength = history.length;
setInterval ("checkHistory()",130);
function checkHistory() {
if (oldHistLength != history.length) {
var o=document.getElementById("outerdivv");
o.style.height ="242px";
}
}
</script>

<div><div id='outerdivv' style="width:540px;
height:74px; overflow:hidden;
position:relative; border: none; ">
<iframe id='inneriframe'
src='http://chat.amyiris.com/widget/'
frameBorder='0' style="position:absolute;
top:0px; left:0px; width:1280px;
height:1200px; border: 1px solid #000000;">
</iframe></div></div>


Stupid Bot Tricks

We're over here trying to advance the field of artificial intelligence for the betterment of the world. And one of the stupid little things we've built is connecting Twitter to a stupid conversational bot. See, we believe that in eight short years, you'll be able to converse with a bot in 140 character bursts, and not be able to tell it's not human. But ya gotta start small. Baby steps.

So in the meantime, we offer this stupid bot.

All we ask is:
  1. Follow @amyiris
  2. Tweet "talk to @amyiris" (then she'll follow you)
  3. dm amyiris, and a bot will converse with you

Tuesday, June 16, 2009

Have Conversations with @AmyIris

The Amy Iris Project is a project to build a better bot. The project is underway, and utilizes the ALICE chat-bot system as a base.

As a very rudimentary demonstration of the Amy Iris System, I've connected the @amyiris Twitter account to the API. Source code is below. So she will reply to Direct Messages, by sending back a Direct Message, if you follow the instructions below.

Here are screen shots of a dialog that I was having with @amyiris while testing. Unfortunately, Twitter doesn't organize this as a back-and-forth dialog, but you get the idea. First, here are the Direct Messages I sent:





And Amy Iris dutifully wrote back to me:






Amy Iris isn't all that well trained at the moment. She will respond mostly with ALICE answers, but it'll give you a chance to play with her API and converse with her. At the time of this publication, she is programmed to use ALICE answers, as well as code snippets that I have discussed in other blog posts: the Best Buy store look-up and the language translation snippets. This is just a rudimentary demo.

To activate @amyiris so that she responds to your direct messages, do the following:
  1. Follow @amyiris (so that you two can communicate). If you already follow @amyiris, you can skip this step.
  2. Send a public Tweet that contain the words "talk to @amyiris". Amy Iris is looking for those words, using Twitter search, and will follow you as soon as she notices (if she doesn't already follow you). You must do step 2, for Amy Iris to answer you in step 3, even if she already follows you.
  3. Send a Direct Message to Amy Iris (example: DM amyiris Hi Amy Iris!)
  4. She will send you a DM response back. Converse as many times as you want.
  5. When you are finished, simply stop conversing with her. Or you leave the test group by "checking out". Simply send a public tweet that contain the words "check out @amyiris". Stupidly clever, eh?

Here are some links to make this process easier:

Click here to JOIN the test group, and talk to @amyiris (this is step 2, above)
Click here to CHECK OUT of the test group. (or just quit sending her DM's.) (this is step 5, above)


Here are some things to talk to @amyiris about:
  • Ask Amy Iris to translate something to spanish or to french. (Example: "Say chair in spanish")
  • Ask Amy Iris where a Best Buy store is in a nearby city or zip code. (Example: "Where is a Best Buy store near 45249?")
  • Or just talk to her. Most conversation will be handled by the ALICE interface, so the responses might sound a bit nonsensical at the moment.

Notes:

  • She's dumb right now - mostly ALICE, plus the translator and Best Buy Store Locator.
  • I've temporarily disabled the code submission portion of the API for now, so you can't make her smarter (yet).
  • Right now she shares one session for all Twitter users. So if one twitter user tells her that his name is Fred, and another user asks what his name is, Amy Iris will think it's Fred.
  • I use Mike Verdone's Python Twitter Tools as my Twitter API. I like this MUCH better than the old Python-Twitter version I was using earlier. You can follow Mike's installation instructions if you want to be thorough. But I think really all you need are two files: api.py and twitter_globals.py, both of which you can view on github at the links provided. Personally, I have those two files, plus a tiny __init__.py file, stored in a folder called twitter, right in the directory with my project, and I am set to go.
  • I heavily borrowed from Mike Verdone's PTT, to create my API. So don't accuse me of ripping it off - I admit it! His GREAT code is licensed to permit this.
  • So that I don't hammer the Twitter servers too hard, I have some delays in the code. If people are actively conversing, I have a 2 second delay in the loop. But if she's fairly idle, then I increase the delay. First to 4 seconds, then 8, then 16, 32, then 40 seconds (max). So if no one's talking to her, you may have an average 20 second delay (which should be fine).

Here's my code:



import time, sys
import amyirisapi
import twitter as twitterapi # this is the Mike Verdone's PTT

# Delays, so we don't hammer Twitter's servers, especially when idle
MIN_PAUSE_INTERVAL = 2.0 # 2 second pause at a minimum
MAX_PAUSE_INTERVAL = 40.0 # 40 second pause, max
PAUSE_INTERVAL_MULTIPIER = 2.0# keep doubling the interval, from min,
# up to max, if nothing's happening
pause_interval = MIN_PAUSE_INTERVAL

# Search Strings
SIGNUP = "talk to" # to start chatting with @amyiris
CANCEL = "check out" # to check out of the test group

SEARCH_TWEET = "@amyiris"



########################################################
# #
# CUSTOMIZE THIS SECTION FOR YOUR USE #
# #
twitter_username="amyiris" #
twitter_password=open("pwd.txt","r").read() #
# #
# Feel free to change to: twitter_password="yourpass" #
# I put mine in a file to hide it from you! #
# #
# END OF CUSTOM SECTION #
# #
########################################################



# twitter api docs request that you put something meaningful into
# the agent string, so they can more easily track usage.
agent = twitter_username + " agent messing with PTT"


# This section initializes a bunch of stuff so I can
# restart the bot later, and pick up where I left off

testers = {} # a dictionary of testers
search_since = 0L # tracks the highest message id searched
dm_since = 0L # tracks the highest Direct Message received

try: #should fail the first time, if the file doesn't exist
chat_track = open("chat_track.txt","r")
testers = eval(chat_track.readline())
search_since = long(chat_track.readline())
dm_since = long(chat_track.readline())
chat_track.close()
except:
pass

# instantiate the APIs for Twitter, Twitter Search, and Amy Iris
twitter = twitterapi.Twitter(twitter_username,twitter_password,agent=agent)
twitter_search = twitterapi.Twitter(twitter_username,twitter_password,agent=agent,
domain="search.twitter.com")
amyiris = amyirisapi.AmyIris()


while True: # loop forever (obviously)

something_happened = False # tracking whether anyone's interacting with
# us on Twitter; start loop at False

# Search for public tweets requesting interaction
try:
search_results = twitter_search.search(q="",
phrase=SEARCH_TWEET, since_id=search_since)
search_since = search_results['max_id']
if search_results['results']:
something_happened=True

for tweet in reversed(search_results['results']):
if SIGNUP in tweet['text']:
print tweet['from_user'],"wants to talk."
try:
if twitter.friendships.exists(user_a=tweet['from_user'],
user_b=twitter_username):
try: #try to follow them so I can receive DMs
twitter.friendships.create(screen_name=tweet['from_user'])
except twitterapi.TwitterError:
pass # hopefully just because we already follow them,
# but possibly because of network / twitter issues
testers[tweet['from_user']]=True
except twitterapi.TwitterError:
pass
elif CANCEL in tweet['text']:
testers[tweet['from_user']]=False
print tweet['from_user'],"finished talking."

except twitterapi.TwitterError:

pass # probably a 403 - Forbidden from Twitter search, due to rate throttling.
# possibly a fail whale or network issue

# Now process and reply to Direct Messages to me

try:
direct_messages = twitter.direct_messages(since_id=dm_since)
if direct_messages:
dm_since=direct_messages[0]['id']
something_happened=True

for dm in reversed(direct_messages):
try:
# Make sure this is a tester first
if testers.get(dm['sender']['screen_name'],False):
amyresponse = amyiris.textin.submit(textin=dm['text'])
twitter.direct_messages.new(
user=dm['sender_id'],
text=amyresponse)
print dm['sender_id'],dm['text']
print amyresponse
except:
pass

except twitterapi.TwitterError:
pass


# Store our placeholder in the file, in case we want to restart

chat_track = open("chat_track.txt","w")
chat_track.write(repr(testers)+"\n")
chat_track.write(str(search_since)+"\n")
chat_track.write(str(dm_since)+"\n")
chat_track.close()


# Pause for a bit, to be kind to Twitter's servers

if something_happened:
pause_interval = MIN_PAUSE_INTERVAL
else:
pause_interval = min(pause_interval * PAUSE_INTERVAL_MULTIPIER,
MAX_PAUSE_INTERVAL)
time.sleep(pause_interval)



And here is the API file (ver 1.0) so that you can build your own interface to Amy Iris. Name this file amyirisapi.py:




# This is Amy Iris API version 1.0, Licensed to all via an MIT license:
#
# Much of the code adapted from Python Twitter Toolkit
# Thank You, Mike Verdone! http://mike.verdone.ca/twitter/
#
# Copyright (c) 2009 Jerry Felix
#
# Portions of this software are
# Copyright (c) 2008 Mike Verdone
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.




DOMAIN = "api.amyiris.com"
POST_ACTIONS = ['submit','register','new','post']

from base64 import b64encode
from urllib import urlencode

import urllib2

from exceptions import Exception

def _py26OrGreater():
import sys
return sys.hexversion > 0x20600f0

if _py26OrGreater():
import json
else:
import simplejson as json

class AmyIrisError(Exception):
"""
Exception thrown by the AmyIris object when there is an
error interacting with amyiris.com.
"""
pass

class AmyIrisCall(object):
def __init__(
self, username, password, format, domain, uri="", agent=None):
self.username = username
self.password = password
self.format = format
self.domain = domain
self.uri = uri
self.agent = agent
def __getattr__(self, k):
try:
return object.__getattr__(self, k)
except AttributeError:
return AmyIrisCall(
self.username, self.password, self.format, self.domain,
self.uri + "/" + k, self.agent)
def __call__(self, **kwargs):
uri = self.uri
method = "GET"
for action in POST_ACTIONS:
if self.uri.endswith(action):
method = "POST"
if (self.agent):
kwargs["source"] = self.agent
break

id = kwargs.pop('id', None)
if id:
uri += "/%s" %(id)

argStr = ""
argData = None
encoded_kwargs = urlencode(kwargs.items())
if (method == "GET"):
if kwargs:
argStr = "?%s" %(encoded_kwargs)
else:
argData = encoded_kwargs

headers = {}
if (self.agent):
headers["X-AmyIris-Client"] = self.agent
if (self.username):
headers["Authorization"] = "Basic " + b64encode("%s:%s" %(
self.username, self.password))



req = urllib2.Request(
"http://%s%s.%s/%s" %(self.domain, uri, self.format, argStr),
argData, headers
)
try:
handle = urllib2.urlopen(req)
if "json" == self.format:
return json.loads(handle.read())
else:
return handle.read()
except urllib2.HTTPError, e:
if (e.code == 304):
return []
else:
raise AmyIrisError(
"AmyIris sent status %i for URL: %s.%s using parameters: (%s)\ndetails: %s" %(
e.code, uri, self.format, encoded_kwargs, e.fp.read()))

class AmyIris(AmyIrisCall):
"""
The AmyIris API class.

Get RESTful data by accessing members of this class. The result
is decoded python objects (lists and dicts).

"""
def __init__(
self, username=None, password=None, format="json", domain=DOMAIN,
agent=None):
"""
Create a new amyiris API connector using the specified
credentials (username and password). Format specifies the output
format ("json" (default) or maybe "xml" in the future).
"""
if (format not in ("json", )):
raise AmyIrisError("Unknown data format '%s'" %(format))
AmyIrisCall.__init__(self, username, password, format, domain, "", agent)

__all__ = ["AmyIris", "AmyIrisError"]


class Code():
"""
Code container for submission of Code Snippets.

Use this class for submitting or reviewing Code Snippets and their meta data.

"""
def __init__(self,**kwargs):

self.agree="checked"
self.rowid="None"
self.author="Anonymous User"
self.publish="Test"

for arg in kwargs:
self.__dict__[arg]=kwargs[arg]






I hope you find this interesting and mildly amusing. I'm sure you can imagine, this is just the beginning. Our goal is to have community contributions to create more powerful Conversational Interfaces.

Feedback, please! And Click Here to ReTweet if you like this post!

Sunday, June 14, 2009

Extending the Amy Iris Python Conversational Interface

In earlier blog posts, I demonstrated a simple Conversational Interface written in Python. In this blog post, I talk about how the responses are generated by the Amy Iris system, and how you can extend Amy Iris to adapt her to your needs, or make her smarter.

Recall, we had this five-line Python program:

import amyirisapi
a=amyirisapi.AmyIris()
for i in range(3):
....kwargs={"textin":raw_input("You: ")}
....print "Amy: "+a.textin.submit(**kwargs)


This simple program prompts You for some input, and submits your input to the Amy Iris system, displaying her response.

Here's a sample execution:

You: Hi, how are you?
Amy: Hello there. I'm doing fine thanks how are you?
You: say I have an emergency in spanish
Amy: tengo una emergencia is i have an emergency in spanish.
You: Where is the nearest Best Buy to 45249?
Amy: The nearest Best Buy store to 45249 is the Fields Ertel OH store, which is 1.47 miles away, at 9871 Waterstone Boulevard.


(Note that the text follows "You:" was typed in by the user, and the text that follows "Amy:" was the system's response.)


The Amy Iris system is built on a foundation of the award-winning open source ALICE chat-bot system. That is to say that anything that you type to Amy Iris is sent to an ALICE program to prepare a response. In the above example, the first response from Amy actually came from ALICE.

The text that you type to Amy Iris also is sent to various code snippets that Amy Iris developers have submitted to the system. Anyone in the world can submit code to Amy Iris for execution. In the second and third examples, above, short snippets were executed to get Amy Iris' answers.

Let's start with a simplified example of how the code snippets work. If we ask Amy Iris about Barack Obama, she doesn't know him, because the Open Source version of ALICE that is the foundation of Amy Iris was written several years ago:

You: Who is Barack Obama?
Amy: I do not recognize the name. I have to process that one for a while.

However, with a simple code snippet, we can make Amy Iris a little bit smarter. After submitting the code snippet to the Amy Iris system, she answers more appropriately:

You: Who is Barack Obama?
Amy: Barack Obama is the 44th President of the United States.

Here's how you can submit a code snippet to the Amy Iris system, using the API. Note, you can also submit snippets through the website, so I am making this slightly more complicated than it needs to be. Stay with me!

Say you write and run the following code once on your PC:

import amyirisapi
a=amyirisapi.AmyIris()
c=amyirisapi.Code(keywordstr="barack,obama",
........license="http://www.amyiris.com/license/AI1.0/")


c.snippet=r"""
if textin=="who is barack obama":
.... say("Barack Obama is the 44th President " +
........"of the United States.",

........confidence=25)
"""

print a.snippet.submit(**c.__dict__)


This program takes a code snippet (a program within the program), and submits it to the Amy Iris system via the Amy Iris API, licensing it to be used freely forever in the Amy Iris system. The code above contains a program within the program. If you run this program one time on your PC, the Amy Iris servers will become smarter, by storing the program within the program. Then, each time someone asks Amy Iris about Barack Obama (in the exact format specified), the code snippet (the program within the program) will be executed, and the answer will be provided.


Let's walk through this, line by line. The first line imports the api, and the second line instantiates the api AmyIris object, so that we'll have programmatic access to the Amy Iris system. The third line instantiates the Code object. The Code object is simply a container to collect all the attributes of the code snippet into one object. These include the Python code itself, some keywords, the title, author, development status, and license. We pass a keyword string, which contains a comma-separated list of keywords for this code-snippet. These keywords help the Amy Iris system determine the relevance of this code snippet, relative to various conversations later.

The next few lines need to be examined as one line. Consider an alternative for a momen, if the line read:
c.snippet="some string"
then it'd be very recognizable to most developers, regardless of your familiarity with Python. But in this case, we use the peculiar but powerful Python syntax of the triple-double-quote pair, essentially accomplishing the same task - to assign a string to a variable called c.snippet:

c.snippet=r"""
if textin=="who is barack obama":
.... say("Barack Obama is the 44th President " +
........"of the United States.",

........confidence=25)
"""


The triple-double-quote pair allows you to define a multi-line string which may contain other quotes. This will make your code snippet readable in your editor. I prefix the opening triple-double-quote with an "r" to make it a "raw-string" meaning that the Python interpreter on your PC will not try to interpret back-slashes escaped characters (although in this case, there are no back-slashes in the code snippet, so the "r" has no impact for this code snippet).

With this single line of code (one logical line, spanning multiple physical lines), the entire code snippet program is stored into c.snippet. That is to say, the "if" statement and the call to "say()" are all placed as one multi-line string into the attribute referred to as c.snippet.

Finally, there's the last line of code of the main program:
print a.snippet.submit(**c.__dict__)
This submits this snippet to the Amy Iris system. c.__dict__ is a dictionary of attributes for c, and that dictionary is passed to the AmyIris object "a", using the method a.snippet.submit(). The api simply rearranges the methods and parameters into a URL that's redefined in the Amy Iris RESTful interface.

The API uses the method name (snippet.submit) to determine that it's a POST to the following url:
http://www.amyiris.com/snippet/submit.json

This RESTful API call will accept code snippets into the Amy Iris system.


The program within the program:


Now let's examine the code snippet itself.

if textin=="who is barack obama":
.... say("Barack Obama is the 44th President " +
........"of the United States.",

........confidence=25)

There are a few tricks and nuances to the Amy Iris system. Amy Iris exposes certain objects, methods and functions to developers' code snippets. Three are particularly significant: textin, say(), and confidence. Your snippet has access to an object called "textin", which is a normalized version of conversation text received by the human. The normalization process makes the text lowercase, removes punctuation and excessive spacing. It also separates sentences and treats them as individual submissions.

The snippet also has access to a function called "say()". "say()" provides a method for your snippet to communicate in the conversation.

One of the parameters of the say() function is "confidence". This parameter allows the developer a chance to state on a scale of 0-100 how confident the developer is in the answer that their snippet is providing. Remember, for a given input, several snippets may be executed (as well as ALICE), and Amy Iris chooses one answer from among the answers that are derived. It's as if Amy Iris is consulting her panel of experts for answers, and asking each expert to give an answer and to state how confident the expert is in that answer. Note that ALICE answers are provided with a confidence<=20, so the developer must specify a number greater than 20 to be sure that Amy Iris over-rides ALICE answers with yours.


Confidence Values:


Before Amy Iris answers the user, Amy Iris will "go ask Alice", as well as execute a number of code snippets to come up with the single best answer. There are a number of parameters that go into Amy Iris' algorithm to determine which answer to give; the developer's confidence rating is one of those. As a developer, it is to your advantage to give an accurate (if not conservative) confidence value. Don't be over-confident.

Here's a guide to confidence values. Please use this guide if you submit snippets to Amy Iris:

Less than 20: Little confidence; ALICE's answer is better.
20-35: Moderate answer, with no examination of the context; probably better than ALICE.
36-50: Answer is definitely better than ALICE, because of a thorough examination of the text submitted, but no examination of the context.
51-75: Pretty good answer - perhaps some examination of the context.
76-99: Near perfect response after examining the text submitted and the context.
100: The single best, most complete answer given the input and the context.

Here are some other factors that may be used in the Amy Iris algorithm, as she determines her response to user input: the code snippet's past success rate, the code snippet's rating by users, and the developer's reputation and rating in the system. So it's in your best interest to use the confidence level accurately, if you want to influence Amy Iris' personality and have your code snippets executed by others.


Summary of Operation:


A user can converse with Amy Iris through a number of methods - Twitter, the web, a program, etc. The user's input is submitted to the Amy Iris server. This server dispatches this input to an instance of the ALICE chatbot software, as well as a number of code snippets. Then the Amy Iris server evaluates the answers that are returned, using a number of factors, including the program-provided confidence value, the snippet's rating, and the developer's rating. A final response is selected and returned to the user.

The software will always get an answer from ALICE, and rates them at a confidence of 20 or less. The concept behind a community developed Conversational Interface is that we, as a community, can improve on ALICE's answers through the submission of snippets. Our vision is analogous to the early days of Wikipedia (eight short years ago): Start with a few example Wikipedia "articles" (or Amy Iris "code snippets"), provide a platform for open source contribution and free access, and allow the Internet Community to develop an intelligent agent that we can all benefit from.

Note that the simple snippet shown in the example above only returns an answer when there is an exact match between the normalized input of the user, and the string in the program. If the user asks "Who is Obama?", this snippet (as is) will not return an answer. In such a case, the ALICE answer (or an answer from some other code snippet) would be selected. Clearly this code snippet could be improved to handle various wordings of the question, and to look up any presidents or world leaders. I leave these as exercises for the reader!

Other blog posts will demonstrate some advanced tricks that you can do with your code snippets. Obviously, not every snippet is going to be of the format "if textin==X: say(Y)". Much more sophisticated parsing and response is possible, using regular expressions for parsing, and the internet to look up responses. Imagine a question about an airline flight looking up the answer on a travel site. Or a question about an actor looking up an answer on IMDB. There are all sorts of interesting possibilities.

Coming up next: The API, so you can get your hands dirty.

It's complicated. Smile if you're gettin' it!
What do you think? Too complicated?

Saturday, June 13, 2009

Technical Explanation of Simple Conversational Interface in Python

In an earlier blog post, I provided a five line Python program that demonstrated the Amy Iris Conversational Interface. The mission of the Amy Iris System is to provide a single working Conversational Interface that developers can easily and freely implement into their websites and applications, that advances the usability of applications, while advancing the state of the art in chat-bot software for customer service and software usability.

I truly feel that Python programmers can do for the field of artificial intelligence what Wikipedia did for general knowledge and encyclopedias. By building a collection of "conversation parsing and processing" into one open source, free resource, an extremely smart Conversational Interface can be collectively built with the wisdom of crowds that makes today's chat-bots laughable.

In this blog post, I walk through the five lines of code, focusing on the Python Language, aimed at users who might not be familiar with Python. In later blog posts, I'll show you how you can put this to work for you, by extending the bot and by embedding her into your applications.

Here were the five lines of code (admittedly using a simple API file that is described later, an in more detail in an upcoming blog post).

import amyirisapi
a=amyirisapi.AmyIris()
for i in range(3):
....kwargs={"textin":raw_input("You: ")}
....print "Amy: "+a.textin.submit(**kwargs)



This simple program prompts You for some input, and submits your input to the Amy Iris system, displaying her response.

Here's a sample execution:

You: Hi, how are you?
Amy: Hello there. I'm doing fine thanks how are you?
You: say I have an emergency in spanish
Amy: tengo una emergencia is i have an emergency in spanish.
You: Where is the nearest Best Buy to 45249?
Amy: The nearest Best Buy store to 45249 is the Fields Ertel OH store, which is 1.47 miles away, at 9871 Waterstone Boulevard.




What's going on technically?

First, let's examine the five lines of Python Code. The first line imports the Amy Iris API. You'll need to download this software to give this a try. The second line creates an instance of the api object, cleverly named "a". The api class is simply an interface to the RESTful Amy Iris System located at Amy Iris.com.

The third statement is simply a "for" loop that will execute 3 times.

The fourth statement sets up a dictionary called kwargs. "kwargs" is a Python idiom meaning "keyword arguments". The concept here is that you can call a function or method specifying a list of arguments, by name, and store them in a dictionary. "kwargs" is a commonly used name for this dictionary, but you could use any valid dictionary name (such as dict1, x, fred, r2d2, etc.).

In Python, a dictionary is an object like a hash table. It stores key-value pairs. And in this case, the key will be the name of an argument (textin, in this case), and the value will be the value of the argument.

So we're prompting the user for raw input, with the prompt of "You: ". And the user can type something in. We will store whatever they key in, into the dictionary with the lookup-key of "textin". The first time through the loop, in my above example, the dictionary "kwargs" holds the value of {"textin":"Hi, how are you?"}. This is how dictionaries in Python typically are represented - in curly braces, with key-value pairs separated by a colon. This dictionary has only one key-value pair.

The final statement calls the api with the arguments stored in kwargs, and prints the results. The "**kwargs" format means that Python should take the dictionary and treat it as a list of parameters, by name/value pair. In other words, the following two lines would be equivalent, the first time through the loop:

a.textin.submit(**kwargs)
a.textin.submit(textin="Hi, how are you?")


So call the api, submitting the user's text to Amy Iris, and print the results.

The api uses the very clever, simplified technique used by Mike Verdone's Python Twitter Toolbox. (As an aside, I was totally impresed by Mike's Python Twitter toolbox - what a great way to access Twitter with Python! Beautiful code! Simple code! I've begun using it instead of the old Python-Twitter API. So, Thanks, Mike, for publishing it as open source!) Similar to Mike's code for interfacing with Twitter (or perhaps as a direct rip off of Mike's code...), The Amy Iris API simply takes the methods used, and passes them along to the Amy Iris RESTful interface.

In other words, the api is called using this line,
a.textin.submit(**kwargs)
The api takes the key-value pairs in the dictionary "kwargs" and submits them via a POST to the url
http://<some server at amyiris.com>/textin/submit.json
[server is defined in the API file]
(note how the api simply rearranges the methods .textin.submit, into the URL of the same name.)

This URL accepts input, runs it through the Amy Iris system, and provides a response. And in this case, it's printed to the user.

This blog post focused on walking through the Python code, so that you can build and implement your own Conversational Interface using the Amy Iris system. Note, if you are not using Python, all is not lost. You can accomplish the same thing in other languages, simply by calling the Amy Iris RESTful interface. Use the url above (that is,
http://<some server at amyiris.com>/textin/submit.json
[server is defined in the API file]) and create software that POSTs to this url, and uses the result that is returned to you (which will be Amy Iris's response to the input supplied in the POST variable "textin").

I'd be interested int seeing how you'd achieve the same access to RESTful interfaces in other languages. I'm specifically interested in seeing implementations in ASP, ASP .net, PHP, Ruby, and as a Facebook app! So if you're up to the challenge, post it in the comments section!

If you made it this far, you may also be interested in reading my earlier blog post about what's going on behind the scenes, and how Amy Iris comes up with her answers. And in upcoming blog posts, I'll describe how you can add Python Code to Amy Iris' logic processing. You'll see how she looks up the language translation and the Best Buy information. You'll learn how you can program her to behave how you want her to for your application.

If you like this blog post, please ReTweet it on Twitter. I'm monitoring the number of clicks on the page, and releasing additional details once people have had a chance to digest the information that I have already published. Thanks!

Friday, June 12, 2009

A Look into Amy Iris' Brain

Amy Iris is a community-built Conversational Interface platform. She responds to user submissions as a chat bot would. This blog post discusses how she determines an appropriate response based on user input.

Users can interact with Amy Iris in chat sessions via the website (amyiris.com), or through widgets on other websites. Web Developers can add an Amy Iris widget to their sites for free, simply by agreeing to the terms, and grabbing the HTML. Developers can also build their own Conversational Interface, using Amy Iris' API. And users can interact with Amy Iris through Twitter.




When a user "talks to" Amy Iris, she seeks out an appropriate response, by submitting the text and any relevant context to various programs.

The first program is called ALICE. It's an award-winning open source chat-bot program that has been around for a few years. ALICE is Amy Iris' "fallback", in case no other developer-submitted code has a response for the user input.

Amy Iris also submits the user text and context to a series of Open Source Contributed Code Snippets, in search of a good response. Amy Iris has an algorithm to select which snippets will be executed, based on a number of factors, including keywords, author's reputation, snippet ratings, success in the past, etc.

Each of these code snippets executes, and many will provide Amy Iris with a potential response for the user, along with a rating stating how confident the code snippet is in the answer provided. The contributing developer will rate their confidence based on a number of factors, such as how thoroughly their snippet evaluated the user input, how much context is used to arrive at an answer, and how good of a fit the answer is.

Amy Iris selects a single answer to be provided to the user based on a number of factors.





The code snippets can run in isolation, or they can contact other websites to retrieve information. For instance, a code snippet written to perform natural language translation (such as English to Spanish) may post to an existing translation website, and parse the results, to present them to the user. This could use an HTML-scraping method, which provides a way to harness all of the information that's available on the web today.

The HTML scraping method is fragile, since the snippet developer is relying on a specific format for the resulting web page. It works, but it can break anytime the website owner substantially reformats their webpage.

This technique should only be performed with the permission of the website owner, and within the "Fair Use" guidelines of copyright law, as you would not want to "republish" information that is not intended for republication.





Code snippets can also interact with APIs (Application Programming Interfaces). For example, Best Buy has an API that they have created called Remix. Best Buy Remix allows you perform lookups on Best Buy's information such as store location, product information, pricing, etc.

APIs typically provide a more robust, sturdy method of information retrieval than HTML scraping, since API specifications are typically published and changes to the specification usually take existing users into account so that they don't break anything.






So, the whole system looks like this. Amy Iris is a repository for developers to submit and manage Open Source code snippets. It's a toolkit to allow you to build "Application specific" bots. It's a collection of widgets that allow developers to integrate the the tools into their own products.






In short, Amy Iris aims to do for Artificial Intelligence what Wikipedia did for Encyclopedic knowledge - provide a free, open source, contributory platform for the benefit of the world.



Presenting at PyOhio July 25-26

One of our team members, Jerry, submitted a paper to discuss Conversational Interfaces at the upcoming PyOhio conference in Columbus Ohio, July 25 and 26. I'd bet he'll touch on our favorite topic, Amy Iris!


We're interested in feedback and ideas for this presentation.

Here's the abstract of the presentation:

Artificial Intelligence and Python:
Developing a Conversational Interface Using Python
Jerry Felix

Conversational Interfaces (CIs) are programs that permit humans to interact with computers using natural language. As the state of the technology evolves from today’s simple chat-bots to tomorrow’s true “strong” Artificial Intelligence beings, Python sits as key component at the heart of these developments. Python has unique advantages as “the development language of choice” for Conversational Interfaces. In this presentation, I will introduce this exciting and rapidly developing field, explain Python’s unique positioning and capability in this area, and demonstrate how users can easily use Python to create their own CI.

Ten lines of Python change the world

One by one, free open source components have emerged that have laid a foundation to allow a novice Python programmer to create very powerful Conversational Interfaces. These include Artificial Intelligence and Natural Language Processing components, APIs such as Python-Twitter, Python and its libraries and Google App Engine. We are now at a point where a novice Python programmer can write code snippets as short as ten lines that can do amazing things, on the shoulders of giants.

Imagine a ten line program that does universal language translation, allowing you to send a text message from any phone in the world, and receive an immediate response:

MMMMMe (texting on cell phone): Translate “I have an emergency.” to Spanish
MMMMCI (texting back to me): “I have an emergency.” in Spanish is “Tengo una emergencia.”


Of course, there are translators available on the internet. So why is this significant? First, it’s pretty cool that with the right ten lines of Python stored in the right place, a novice can turn every cell phone on the planet into a universal translator. But that’s just the beginning.

All it takes is a little imagination and extrapolation to see the greater significance: If a novice programmer can write ten lines of Python to do something pretty cool, what happens when tens of thousands of novices realize this, and begin writing and contributing to a central repository of code snippets? Something Really Cool: a universal Conversational Interface emerges. What Wikipedia did for Encyclopedias, Python programmers can do for Artificial Intelligence.


Presentation Contents:

During this presentation, I will introduce Conversational Interfaces, and the components necessary to create your own CI using Python. I’ll explain Python’s unique role in Artificial Intelligence and CIs. I’ll demonstrate building code snippets, walk through the simple logic, and show how attendees can write similar code to respond intelligently to various conversational inputs. Then I’ll talk about how anyone can use Conversational Interfaces to improve their websites and Python applications.

Expertise Level Targeted:

This presentation includes discussions of very small code snippets which are appropriate for novices and experts alike. However, due to the subject matter, it should not be considered as an introduction to Python for someone who has never programmed before. You may consider it a good candidate for the Beginner’s Track because it shows what’s possible with very little Python code, and may open the eyes and minds of some beginners. The material will be presented in a fashion such that beginners will not be lost, but intermediate to experts will likely grasp it more fully.

Area of Python Programming that it relates to:

This presentation relates to the following areas of Python Programming: Libraries (especially re, urllib, urllib2), string parsing, interfacing with APIs (such as Python-Twitter, and the Best Buy ReMix API), internet programming, Google App Engine, Artificial Intelligence, and Conversational Interfaces.

About the author:

For 16 years, Jerry was employed by Hewett-Packard as a Systems Engineer and a Manager of Technical Consultants supporting major accounts including P&G, GE, Walmart, GM and others. He has participated in the building and support of the programs and technologies at some of America’s largest corporations. During the past 13 years, Jerry has been the co-owner and executive of a 15-person Cincinnati-based software business, Electronic Commerce Link. Jerry has spoken at international conferences before, including the HP International Users Group and the P&G Worldwide Users Conference. He holds a BS in Systems Analysis from Miami University and a MBA from the University of Cincinnati. Jerry is a self-taught Python programmer and enthusiast.

Creating a Conversational Interface using Python

Conversational Interfaces (CIs) are programs that permit humans to interact with computers using natural language. Conversational Interfaces are evolving from the simple "chat bot" and increasing in intelligence toward the realm of truly helpful artificial intelligent agents.

The Amy Iris project is a toolkit that facilitates the creation and usage of a central repository of Conversational Intelligence. In this blogpost, I show you how you can use the Amy Iris toolkit to create a Conversational Interface for your website or software.

Let's start with a demonstration of the power and simplicity of the Open Source Amy Iris API. For this example to work, you must install the Amy Iris API (one file, open source), which will be explained in another blog post. Yes, this is another "teaser" blog post... sorry!

Here's a sample program:


import amyirisapi
a=amyirisapi.AmyIris()
for i in range(3):
....kwargs={"textin":raw_input("You: ")}
....print "Amy: "+a.textin.submit(**kwargs)

This simple program prompts You for some input, and submits your input to the Amy Iris system, displaying her response.

Here's a sample execution:

You: Hi, how are you?
Amy: Hello there. I'm doing fine thanks how are you?
You: say I have an emergency in spanish
Amy: tengo una emergencia is i have an emergency in spanish.
You: Where is the nearest Best Buy to 45249?
Amy: The nearest Best Buy store to 45249 is the Fields Ertel OH store, which is 1.47 miles away, at 9871 Waterstone Boulevard.



(Note that the text follows "You:" was typed in by the user, and the text that follows "Amy:" was the system's response.)

Pretty cool simple little bot in five lines of code. The Amy Iris bot is an extended version of the award-winning ALICE chat-bot, built with Open Source components. In the above example, Amy's first response was the exact response from Alice, when presented with "Hi, how are you?"

But the second and third responses hint at the potential power of Amy Iris. The second response, the language translation, and the third response, the Best Buy store look-up, are both the results of a small open source code snippets that were submitted to the Amy Iris system by someone around the world (in this case, me!)

Amy Iris provides a central repository for open source code snippets written by developers from around the world. It's a community-developed bot: the wisdom of crowds boiled down into a conversational interface.

In an upcoming blog post, I'll tell you how this all works! Gotta keep ya coming back, right?