Qatar’s emerging technology scene

Every now and then a senior year student asks me “Do you know of any companies that are hiring computer science graduates?” Or the parent of a prospective student asks me, “Should I let my daughter study Information Systems, is there really a market for this major?”

I try to answer with as much enthusiasm as I can gather. I tell them about the tons of internships students do each year. I tell them about myself, about how I got myself into General Electric. As long as I talk, I can see a glow in their eyes. But it is very short-lived. I know as soon as I leave, they’ll start to have doubts again. I wish I could point people to a resource that they could browse at the comfort of their homes and use to make informed decisions about themselves or their children.

I believe there is a big gap between what the citizens of Qatar believe about the job market, and how it really is. And it’s really a matter of shame. Qatar has been investing in its knowledge based economy for about a decade now. To make this investment worthwhile, we need to close this gap from both sides. We need a platform to let students know what’s happening in Qatar’s technology scene, and we need the tech-companies to know more about students, their curriculum, their research, their projects etc..

All this is about to change with “Emerging Technologist”, a one of a kind internet show aimed at bridging the gap between Qatar’s tech industry and its world-class academia. Over the course of the next few months ET will:

  • Chase down working graduates from leading universities in Qatar and interview them about their jobs, their experience and their ambitions.
  • Collaborate with professors to get some insight on academic research and have students describe their contribution.
  • Interview potential employers and build detailed job profiles ahead of the career fair season.
  • Maintain up-to-date records of internship and full-time employment details of students majoring in tech.
  • Go on a university-level talent hunt looking for budding technology entrepreneurs with interesting personal projects.

The show is currently going through the planning phase for season 1. The current team consists of recent graduates, seniors and juniors from Carnegie Mellon University in Qatar. To be able to produce this show successfully we’ll need talented students to direct, record and edit our short episodes. The team is also looking for student contacts in TAMUQ, QU and CNAQ to expand our coverage to all technology domains.

If you are interested in a semester-long engagement, or willing to brainstorm with us over the summer, drop us an email, or follow us on:

emergingtechnologist@gmail.com

http://emergingtechnologist.tumblr.com

Programmer’s Wish List – 2

Multi-column Code Editor

Imagine…

  • An IDE with multi-column editor. Each of them are synced. Adding a new line in one adds it in all of them.
  • Use one column for source, one for documentation, one for writing tests, one for logging.
  • Ability to add more columns to define functionality that you want to execute but don’t want to mix with the main algorithm.
  • Weave the code from different columns at compile time, or use byte code instrumentation to combine the sources.

Web Powered Unix Terminal

Imagine…

  • A full-blown Unix terminal rendered inside a browser tab.
  • Being able to run “shell-scripts” written entirely in JavaScript.
  • Besides textual output, scripts will be able to take full advantage of the terminal DOM and display graphical and interactive output.

Programmer’s Wish List – 1

Declarative Language for Data Dependency

Imagine…

  • An XML file where you can define the data dependency graph of your application.
  • A library that reads in this graph and executes a chain of methods taking advantage of parallelization whenever possible.
  • A common set of interfaces which can be implemented to define to job distribution, serialization, and load balancing for any execution environment.

Mathematical Expression Injection Framework

Imagine…

  • An XML file where you could define mathematical expressions and map variables to properties of Java beans.
  • A library that reads in XML file and triggers the evaluation of a list of expressions as a response to an event.

The Conversation Model in Interaction Design

As you can probably tell by the title of the blog, Savant is a program that lets users interact with it in natural language and accomplish a large variety of tasks. However, I am guilty of coming up with a title that is quite misleading. Natural Language Processing is a very difficult problem in computer science and I am, by no means, claim to have solved it. The syntax for Savant is quite flexible and allows users to specify arbitrarily complex queries. The rules for Savant’s grammar are very intuitive, and occasionaly form gramatically correct English sentences. But my no means it is a complete natural language parser. For example, you could tell Savant, “move doc files to new folder Spring09″. But you couldn’t say, “move doc files to new folder called Spring09″, or “move files with doc extension to a new folder called Spring09″.

One can argue that all this is syntactic sugar and the grammar can be easily extended to parse these commands. But is the right approach to solve the problem? Here are new a couple of other ways we could go about to tackle this issue. The first method involves filtering non-essential words. Note that in two alternate sentences in the above example, the main key words were already there. Just by filtering out a few words, we can realize the original intent of the command. The second method involves richer interaction with the user as he is typing in the command. As the user types ‘mov’ we can lookup a list of commands and find out the list of commands that start with that prefix. Then for each one of those commands syntax suggestions can appear to guide the user to input his command in a way that the program understands. A positive feedback loop can be started by rewarding the user with auto-completion and detailed syntax break-down. On the contrary, the program can stop suggesting when the user starts going off track, creating a negative feedback. This helps in keeping the core parser simple and still allow flexibility for the user.

If you stop to think for a minute and imagine Savant as a human being, the latter method sounds like a dialogue between two people. The positive/negative feedback is something we constantly provide with various facial expressions. Think about it, how many times has your teacher re-explained a topic after seeing a blank face? How many times have you had the other person complete the word that you just couldn’t think of? Can you see how these real life scenarios have direct mappings to the interaction between man and software. In fact, I believe any kind of human-computer interaction can be explained in terms of a conversation between two people of different languages.

Here are two scenarios to illustrate my point. Think of the command line from UNIX or DOS. To be able to use the terminal you need to memorize the names of the commands and the correct sequence of parameters they take as arguments. So, how does this scenario look in the Conversation Model? It’s similar to an Englishman learning Japanese to talk to person from Japan, or vice versa. Now, learning Japanese can be a very hard prospect. Especially for people who are geographically far apart from Japan. But it is no doubt that the best way to communicate with a Japanese person is in Japanese. That’s the language that he is most comfortable and you can express very complicated ideas, and he’d still be able to understand. This is the case with learning the command line way. The learning curve is tremendous. But those who have mastered it are at the peak of productivity and can express intricate tasks with very concise notation.

The second scenario is of the point-and-click interfaces that we are so used to these days. To make another analogy, this is similar to Sign Language. By sign language I dont mean languages like ASL, but the most basic hand gestures and facial expressions. The beauty of these gestures are they are universal. Irrespective of what language the other person speaks, you can always point at your wrist and indicate that you want to know the time, or nod your sideways to say no. The drawback with this system that it only works with basic things. Try asking somebody the directions to the closest museum using sign language. It is apparent that it is not the most of expressive of language. Some tasks were never meant to be described using sign language, unless you begin a new convention involving careful hand gestures. The problem with GUI is precisely this. Point-and-click makes sense for a handful of things. For others, it becomes a long repetitive process of pressing boxes and ticking checboxes.

This brings us back to our conversation with Savant. The motivation of the project was to not require the user to memorize Savant’s vocabulary. Instead, Savant could master the human language, which would require solving the natural language processing problem. Or, the other option for Savant would be to have a conversation with the user and work with him to come up with something that it can carry out as required. Making this conversation efficient does not require a technological break-through. However it does require pro-active involvement from Savant’s part to guide the user in typing his command.

Savant: The Natural Language Shell

How many times have you stared at the contents of a folder for 5 minutes trying to carefully select the files that you want to copy to your thumbdrive? Maybe you want to remove all the zip files that you downloaded off the web today. Or maybe you want to make a backup of all the folders matching the name hwX where X stands for a number. Sure you can do these things with the file manager of your choice.

The most naive user would go and manually select every one of the 20 items he is thinking of. In the process, maybe, he clicks an empty area on the screen and his whole selection goes away. Since there is no UNDO feature in file managers, he has to start over again. (Makes me wonder why hasn’t anybody thought of this).

A more average user, or more likely, a user with a very clear query in his head will probably use the search feature built in to most file managers. The user fills in numerous checkboxes, fills in the details of whatever he remembers about his query. Hopefully, if the search is narrow enough the right results show up and the user is happy. However, more often that not, using the search feature can be a bit tedious. Ask yourself, how likely are you to use search when you want to copy, delete, move or zip more than 10 files.

On the other end of the spectrum, the macho command line geek will probably laugh at this problem. He will promptly bring up a terminal window and type in some obscure commands involving ls, rm with a bunch of regular expression arguments along with -l -u and what not. If the query is a bit more complicated, and the user is intellectually challenged, he will write a shell script to do the job for him. However, the truth is, on average cases, the command line geek will get his job done faster than the other user groups.


No, I am not advocating the use of command line in our daily lives. But, do you notice that, in a way, the more technologically inclined you are, the faster you can get things done. If you are not so good with computers you will probably end up following medival ways of doing things. However, getting things done fast doesn’t come without a cost. As a user, you are trading off intellectual effort of command line trickery for boring and repetitive chores. Unfortunately for most users, there is no trade-off to make. Such users resort to the latter way of doing things.

So is there a sweet spot between tasks of high intellectual effort and the brainless and repetitive ones? Personally, I think so. In this article I will try to explain what motivates me to think in such a manner.

To put things into perspective I will zoom out a bit more and talk about a more fascinating trend. Back in the days of the first browsers, the address bar was a place where you just typed in the actual domain name. Search engines were not so popular back then. So it was important that you remembered the web address precisely. As browsers got more advanced they started packing more juice into the address bars. First came the drop down of addresses from the history. If you have visited a site in the recent past, all you have to remember is the first few characters of the domain name. A significant reduction in intellectual effort, a new potentially brainless chore. With the advent of Firefox 3, the address bar became even more “awesome”. You could get away with typing only a part of the actual domain name, page title, or even tag name. This new feature quickly spread like wildfire and was adopted by every other browser in the market. With the smarter address bar, daily browsing reduced to typing in a few key strokes. Google Chrome took this even further by displaying Google search results as you type in.

Now, I ask the question again. Do you see a trend. With every new addition, we are actually accomplishing a much more complicated task. We started of by pointing the browser to a specific location on the web. We then progressed to doing a domain name search on our recent browsing history. That got elevated to performing a string search of titles, domain names, bookmarks, and tags from the history. Finally we ended up performing an actual web search on Google along with everything else. However, the actual intellectual effort of performing these increasingly complicated didn’t go up that much. It is still pretty much limited to a few key strokes that we easily get used to.

So the moral of the story, it is possible to do complex tasks without doing much. Which brings us back to the issue of selecting, copying, and moving files with your file manager. The current state of the art approach to this problem is to use the sort and search functions of the manager to set things up to do whatever you want. Imagine if you had to do the same with web browsers. Instead of the address bar auto-completing recently visited sites, what if you were shown a list of all pages you have visited recently. Then you had to sort that alphabetically, or by recency, and then perhaps do a search to find what you are actually looking for. Fortunately browser UI designers have been clever enough to spare us the horror of going through all that.

So you see, the way we do things in a file-manager is pretty silly. We somehow fail to realize how silly it is until an analogy is shown to us. What if we could tell the file manager what we want? What if we could tell: “Delete all the zip files created today”. Or maybe, “Move all files starting with hw to Documents/School/”. Command line people may instantly conjure up the appropriate shell commands for each of those tasks. And that’s exactly my point, we need something as powerful as the command line, but for average users. We need something that can understand commands given in natural language and perform them. I’m not talking about having a conversation with the computer. We just need to be able to give domain specific commands in natural language.

Savant, is a proof of concept of such an idea. It is a natural language shell to perform routine tasks that we do using a file manager. It is simple in the sense that it can perform only a handful of tasks, but it is also powerful in the sense that it can perform those tasks on any complicated query that we describe in natural language. What’s more exciting about this idea is, Savant need not be a solution for a handful of problems. It can be as extensible as Firefox. Developers should be able to extend the program to do more interesting tasks. Alongside, they can also specify the natural language commands using plain old context free grammars. To make parsing easy, the front-end can suggest syntax as the user types in his command. This way, it is clear for the user what the system can and cannot understand.

Sounds interesting. Stay tuned for more updates…

Snowfox: The Semantic File Browser

So how many files and folders of your computer do you interact with? A small fraction only. The typical things we deal with everyday are office documents, home-work assignments, pictures of families and friends, music. Those of you who are technologically inclined have code written in variety of languages saved on your computer. Often you also have text-snippets storing phone numbers, random links, a chat log of a friend, an interesting idea.

Ever noticed that your file manager never bothers about what the contents really are. For all its purposes, they are just files, folder, shortcuts etc. An average file manager does not distinguish between system.ini and historyEssay02.doc, other than opening it with different applications. Its upto the average Joe to be smart and organize his “files” into folders, sub-folders, sub-sub-sub folders, meaningful file names, meaningful locations etc.

Here’s a question, why should file browsing be so radically different from web browsing. When you visit a site, are you ever offered a list of file names ending with html, php, asp, pl ? How do you visit a website? Do you have to go through a heirarchy of directories to reach your favourite sites? Or do you just type the name of the site, or part of it in the address bar?

One more question, how does your browser home page look like ? Does it say C:, D:, DVD-RAM Drive E: ? It makes perfect sense to have a google search box, a personalized start page, or even a blank page as your home page. Why is it so difficult to imagine such ideas for file browsing. Why can we tag feeds and label emails but do nothing with files and folders. Why does the web guide us through words, images and links and file managers leave us with nested heirarchies of data? Who invented the file-tree anyway ?

Snowfox, is a concept of a new paradigm of file browsers. It aims to contextualize the contents of your computer. It frees you from the shackles of heriarchies and lets you visually set up contexts for your files. With contexts the browser changes itself to adapt to your needs. Snowfox is a productivity tool. It lets you get to your “stuff” quickly and more meaningfully. Its more of an organizer than a mere browser. It brings modern tools like tagging, bookmarking, scripting and makes them a part of your user experience.

Nutshell in a nutshell

Without knowing anything about Nutshell, guess what the following command outputs:

If you guessed that it displays only those files in the current directory that are less than 2000 bytes in size, that would be correct. But that wasn’t too hard, was it? The syntax is quite intuitive. Now compare it what you would really have to type in bash:

Show this command to a seasoned Unix guru. Even he can’t decipher this command without looking at the output of ls -l first.

So what makes performing the same command in Nutshell more intuitive. Well, first of all, Javascript has more syntactic sugar than bash script. Secondly, there is something really sweet going on behind the scenes. When you chain two unix commands together with a pipe, the textual output of the first is passed on as an input to the second. But if the second command is to do anything useful with this input, it has to go and parse it first. That’s what the awk command is doing. It is splitting the output of ls -l ,taking the 5th token of each line, performing the comparison, and printing the whole line if the test succeeds. This is where shell scripting is unproductive. It’s all about parsing output.

So how does Nutshell solve this? By using a technique called object piping. When you type in ls -l in the Nutshell prompt you are actually running a Javascript wrapper of the same unix command. This wrapper does all the hard work of parsing the output for you. But in addition to that it also creates and returns an object representation of the output. When you pipe together commands in Nutshell, the successive commands recieve the piped object. So all you need to do then is access the desired fields of the object and produce your own result object. No more parsing to deal with. The ls command outputs an object which looks like this:

All the filter command needs to do is evaluate the function passed to it as an argument, run it over the list, and call the toString over each object. Simple, intuitive and saves a couple of hours of hunting though man pages of awk and find. The aim of Nutshell is to be fully compatible with bash. So you can also use traditional Unix commands in a pipe. Objects are turned into strings by calling their toString methods before being passed to a Unix command. So, you are into command-line masochism, you can write commands like this in Nutshell:

If you like the idea of an object oriented shell with Javascript as the scripting language. Grab a copy of the source from Github and give it a test drive.

Annotated Event Listeners

After my initial work on Swing Query, I was reasonably happy with the results. Even though SQuery covers only a handful of the selectors offered by jQuery, it does a remarkable job of cutting down the amount boilerplate code needed to get a GUI up and running.

The next area I wanted to tackle was the event handling mechanism in Swing. Whenever I write event listeners I always face a dilemma. Should I use:

  1. Anonymous inner classes
  2. Make the parent class implement the appropriate listener interface.

Personally I go for the second option because it lets me write event handlers as normal methods and not have me worry about access restrictions of anonymous classes. This is what I am talking about:

Anonymous classes give us the advantage of defining the listener inline. But you can no longer access member variables in a straight-forward manner. Plus there’s also the disadvantage of the avalanche of nested curly braces. Implementing the listener interface does not have these problems, but it has its own quirks. For example, once a parent class implements a listener interface, it is a natural thing to point all your button ActionListeners to this.

This way of sharing a single event listener forces you to write if conditions to tell the sources apart.

All these if conditions make your code harder to read and debug. So how can we have the best of both worlds? Enter, Swing Query and Annotated Event Listeners. If you read my previous article, you’ll remember that Swing Query let’s add ids to any Swing component.

Building on top of this idea, Swing Query now let’s you annotate ordinary methods with a query and an event name and turn them into event listeners.

There you have it. The sweet spot between anonymous classes and implementing an interface. You get to add event listeners inline without a zillion nested curly braces. Plus the listeners are all normal methods with access to member variables without any additional hassle. Just annotate your methods, and Swing Query takes care of the rest.

Introduction to Swing Query

I make GUIs for a living. I spend weeks at a time, pulling out JComboBoxes and JTextFields from the UI treasure-box called Swing and slap them over my JFrames and JPanels. And I’ll be honest: It sucks. Not the designing UI part, thats fun. But writing the thousands of lines of boilerplate Swing code to get things to show up on the screen. After doing this for 2 years, I told myself, just because I am required to code corporate GUIs in Java, doesn’t mean I can’t have fun while coding. Why should JS coders have all the fun?

So one weekend, out of utter frustration, I coded up Swing Query, a Java port of the jQuery library for Swing. For those of you don’t know what jQuery is, shame on you. But it lives up to its tag line: “Write Less, Do More”. That’s precisely what I wanted to do. Given the general verbosity of Java code, I wanted to write a lot less and, given my requirements, I wanted to do much more. So I began writing the library to ease my wounds, one scar at a time.

For the purposes of this blog post, I’ll be using an example that I stole from the Swing Tutorial site from Sun (yikes Oracle).

Now have a look at the following snippet, and tell me you have seen this a thousand times before:

I don’t know about you. But this particular way of coding always bothered me. Why do we have to put these lonely variable declarations up in beginning of a class definition. You might say, “Maybe you need to access them outside of the method they were created in, like in an event handler, perhaps?”. Spot on, fellow sufferer. These variables are indeed used in an ItemListener.

If you think about it. It’s kinda lame that you have to put a variable in a class-wide scope. All you want is to check which component fired the event? That is certainly not worth all the pain. Also, formal scopes for components like buttons and checkboxes don’t make sense at all. Even if they are declared locally inside a method, you can still have access to them if you get hold of the parent container.

So, let’s give these checkboxes some good old fashioned CSS-ish ids. So we can refer to them by their id names instead of creating unnecessary global variables.

Let’s assume there’s an object called squery that is responsible for keeping track of all the children of the parent container and their ids. We can now use this object to give ids to the checkboxes:

Not a great improvement in terms of number of lines written. But this is a war against a source of personal frustration. At least there are no global checkboxes any more !! Time to move on. What else do I hate to write? Ah yes, I hate typing chinButton.setThis and chinButton.setThat and keep on repeating that for every checkbox. If wish I could say it all at once. Like, I need a checkbox that is selected initially and its mnemonic set to the character ‘c’.

Now with sQuery, you can get away with writing this:

Again, I don’t know if this means much to you. Its a lot easier to chain these setters together if I only have to set a few things for every checkbox. Now we have begun to improve the line count of the program. Let’s see if we can do better.

See if you can identify a common pattern in these two snippets:

In my mind, I’m thinking, “I need to add the same item listener to all these checkboxes, and then I need to add them to the panel”. But in reality, I can’t deal with these 4 checkboxes as a whole. So i need to add the ItemListener separately and add them to the panel separately. Pssh.. Wish I could some how group together these 4 checkboxes and say, the group as a whole needs this ItemListener, and as a whole it needs to be added to the panel.

Now, isn’t that handy. We added an extra unit in our chain. But that saved us a total of six lines. We can do even better. All the 4 checkboxes have their selected property set to true. No point of saying that 4 times.

At this point, the perfectionist in me kicks in. If we are going to use ids and group names, then we don’t need to assign the checkboxes to a variable at all. That’s true. We can now say:

It just keeps getting better. Now reduced 4 more lines of variable declarations at zero cost. Let’s get back to the event handler.

So why do we need those ids again? Oh, because we want to know which checkbox was clicked, and set the values of index and ‘c’ accordingly. Only if these values were embedded inside the checkbox itself, we wouldn’t need to have all those if statements. But if that makes you think of sub-classing the JCheckBox class, or having arrays of CheckBoxes, indices and the characters, then I feel pity for you. All we need is to associate some data with the check boxes.

I know the chain is getting a bit long. But note that there are two thing happening here. I got rid of the ids. I just anonymously add the checkboxes to the “choice” group. And I use the data method to associate a key,value pair with every checkbox. The end result is a simplified event handler.

Now that just shaved 12 more lines from our code, leaving behind a perfectly readable event handler.

Hope that made those nasty Swing burns of go away. There’s no reason Swing development should be as tedious as it should. It’s a very useful cross-platform GUI toolkit. Why not make it more useful and using a library that makes it fun to use.

Get your copy of SQuery today. Or even better, contribute to the project. Happy coding.