Limitations

First of all, why are there any limitations at all if the Phone’s platform is Silverlight and IronRuby works on Silverlight since its inception? Although the UI and Media APIs are pretty much the same the Phone differs from the browser plug-in in the CLR execution engine and the Base Class Library. The Phone uses Compact Framework (CF) which provides only a subset of functionality available in the browser. The two platforms should converge in the future and the limitations will hopefully go away.

The main feature missing from the CF is Reflection.Emit. There are other differences but this is the most important one concerning dynamic languages. As you can imagine we do emit some code at runtime to implement dynamic features of our languages. However we also have an interpreter around. Each DLR based language can choose when to use the interpreter and when the compiler. Both IronRuby and IronPython interpret user code that is executed only a few times. It’s not efficient to compile such code since the cost for doing so is high and the execution throughput is not important. On the other hand code that’s running many times should be compiled and optimized. If the interpreter finds out that a function or a loop body is being executed more than N times it spawns its asynchronous background compilation. When the compilation is done the instructions that call the function or enter the loop are replaced by new ones that jump to the compiled code. The threshold N is configurable.

It should be clear now how to work around the lack of Reflection.Emit on the Phone. Let’s just interpret all the time! And indeed it mostly works. You won’t get a great throughput performance out of it but it works just fine for common scripts. Since the CLR interop is still available you can write performance critical code in C# and call it from the scripts. There are only a few CLR features that you need to avoid since the current interpreter doesn’t handle them without resorting to Reflection.Emit. The most significant are calls to methods with out or ref parameters. You also won’t be able to inherit a Ruby class from a CLR class or implement a CLR interface since that requires us to emit a proper CLR type. We are going to address both of these limitations in future versions.

So, IronRuby runs on the Phone. Why IronPython doesn’t yet? There is no technical reason why it couldn’t. Some parts of the IronPython runtime were just written before we implemented the interpreter and thus need some refactoring to enable full interpretation. You can motivate us to do the work faster by voting on CodePlex.

Sample App

Let’s write an app that hosts IronRuby on the Phone. It’s really simple! You just use the DLR Hosting APIs as you would do when embedding a DLR scripting language in a Silverlight app. I’ll show only the interesting parts of the app. You can download the full source code here. You’ll need Visual Studio 2010 for Windows Phone CTP and IronRuby 1.0 RC4 to try it out. Install them both and then open PhoneScripter.sln, build the solution and deploy to the emulator. Make sure that references to IronRuby and DLR assemblies in the project are correct.

The few lines that you need to run a Ruby script in the Phone emulator (and hopefully on the real device when available) are:

• _engine = Ruby.CreateEngine((setup) => { setup.Options["CompilationThreshold"] = Int32.MaxValue; });

  Creates a Ruby engine that never compiles the interpreted code. I guess IronRuby could max out the compilation threshold by default when running on the Compact Framework. For now you need to do this manually.

• _engine.Runtime.LoadAssembly(typeof(Color).Assembly);

  Loads System.Windows assembly into the dynamic runtime so that we can script the UI. System and mscorlib assemblies are loaded by default.

• RubyContext context = (RubyContext)HostingHelpers.GetLanguageContext(_engine);
  context.ObjectClass.SetConstant("Phone", this);

  Makes the MainPage class instance (this) available to the script via a global constant Phone. We’ll use it to access UI elements on the page. This is a hack! You should use ScriptScope Hosting API to expose host objects to the script in your .NET apps. However, the scope dynamic object internally uses methods with out parameters and we can’t interpret calls to it from Ruby. We’ll enable this in future. For now, we work it around by directly accessing IronRuby’s RubyContext class. Be aware that this class is not a part of the Hosting APIs, we might change it in future versions and break your code. So do not use it in any production code.

• MemoryStream stream = new MemoryStream();
  _engine.Runtime.IO.SetOutput(stream, Encoding.UTF8);
  
  try {
      try {
          _engine.Execute(Input.Text);
      } finally {
          byte[] bytes = stream.ToArray();
          Output.Text += Encoding.UTF8.GetString(bytes, 0, bytes.Length);
      }
  } catch (Exception ex) {
      Output.Text += ex.Message;
  }

  Executes a script entered in Input TextBox, captures its output and appends it to the content of Output TextBox.

We can now enter and run a Ruby script:

The ForwardingMetaObject class defined below makes it easy to forward dynamic operations from one dynamic class (forwarder) to another (forwardee).

public sealed class ForwardingMetaObject : DynamicMetaObject {
    private readonly DynamicMetaObject _metaForwardee;

    public ForwardingMetaObject(Expression expression, BindingRestrictions restrictions, object forwarder, 
        IDynamicMetaObjectProvider forwardee, Func<Expression, Expression> forwardeeGetter)
        : base(expression, restrictions, forwarder) { 
       
        // We'll use forwardee's meta-object to bind dynamic operations.
        _metaForwardee = forwardee.GetMetaObject(
            forwardeeGetter(
                Expression.Convert(expression, forwarder.GetType())   // [1]
            )
        );      
    }

    // Restricts the target object's type to TForwarder. 
    // The meta-object we are forwarding to assumes that it gets an instance of TForwarder (see [1]). 
    // We need to ensure that the assumption holds.
    private DynamicMetaObject AddRestrictions(DynamicMetaObject result) {
        return new DynamicMetaObject(
           result.Expression,
           BindingRestrictions.GetTypeRestriction(Expression, Value.GetType()).Merge(result.Restrictions),
           _metaForwardee.Value
       );
    }

    // Forward all dynamic operations or some of them as needed //

    public override DynamicMetaObject BindGetMember(GetMemberBinder binder) {
        return AddRestrictions(_metaForwardee.BindGetMember(binder));
    }

    public override DynamicMetaObject BindInvokeMember(InvokeMemberBinder binder, DynamicMetaObject[] args) {
        return AddRestrictions(_metaForwardee.BindInvokeMember(binder, args));
    }

    public override DynamicMetaObject BindConvert(ConvertBinder binder) {
        return AddRestrictions(_metaForwardee.BindConvert(binder));
    }

    // ... //
}

Let’s use this class in an example. Let’s define some class B simply as a subclass of DynamicObject that supports GetMember, InvokeMember and Convert to String dynamic operations:

public class B : DynamicObject {
    private readonly string _name;

    public B(string name) {
        _name = name;
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        result = "got member " + _name + "." + binder.Name;
        return true;
    }

    public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result) {
        result = "invoked member " + _name + "." + binder.Name;
        return true;
    }

    public override bool TryConvert(ConvertBinder binder, out object result) {
        if (binder.Type == typeof(string)) {
            result = _name;
            return true;
        } else {
            result = null;
            return false;
        }
    }
}

And now let’s define a class A that forwards these operations to B via ForwardingMetaObject:

public class A : IDynamicMetaObjectProvider {
    private readonly B _b;
    public B B { get { return _b; } }

    public A(B b) {
        _b = b;
    }

    DynamicMetaObject IDynamicMetaObjectProvider.GetMetaObject(Expression parameter) {
        return new ForwardingMetaObject(parameter, BindingRestrictions.Empty, this, _b, 
            // B's meta-object needs to know where to find the instance of B it is operating on.
            // Assuming that an instance of A is passed to the 'parameter' expression
            // we get the corresponding instance of B by reading the "B" property.
            exprA => Expression.Property(exprA, "B")
        );
    }
}

Finally, let’s run some C# 4.0 code that shows how it all works together:

class Program {
    static void Main(string[] args) {
        B b1 = new B("b1");
        B b2 = new B("b2");

        dynamic a1 = new A(b1);
        dynamic a2 = new A(b2);

        Console.WriteLine(a1.SomeMember);
        Console.WriteLine(a2.SomeMember);
        Console.WriteLine(a1.SomeMethodCall());
        Console.WriteLine(a1.OtherMethodCall());
        Console.WriteLine((string)a1);
    }
}

Output:

got member b1.SomeMember
got member b2.SomeMember
invoked member b1.SomeMethodCall
invoked member b1.OtherMethodCall
b1

The really great thing about DLR is that your dynamic objects will also work in IronRuby and IronPython even though you haven’t thought about it at all when authoring them. Assuming that we compiled the above code into MyDynamicLibrary.dll we can run IronRuby REPL and check it out:

IronRuby 0.9.1.0 on .NET 4.0.20915.0
Copyright (c) Microsoft Corporation. All rights reserved.

>>> require 'MyDynamicLibrary.dll'
=> true
>>> b1, b2 = B.new('[b1 in Ruby]'), B.new('[b2 in Ruby]')
=> [B, B]
>>> a1, a2 = A.new(b1), A.new(b2)
=> [B, B]
>>> a1.SomeMethodCall
=> 'invoked member [b1 in Ruby].SomeMethodCall'
>>> a2.OtherMethodCall
=> 'invoked member [b2 in Ruby].OtherMethodCall'
>>> a1.to_s                             # invokes ToString
=> "B"                                   
>>> a1.to_str                           # invokes dynamic to string conversion
=> "[b1 in Ruby]"

Let’s get right to the final REPL code:

# github: repl.rb
load_assembly 'Microsoft.Scripting'
Hosting = Microsoft::Scripting::Hosting
Scripting = Microsoft::Scripting

class REPL
  def initialize
    @engine = IronRuby.create_engine
    @scope = @engine.create_scope
    @exception_service = @engine.method(:get_service).of(Hosting::ExceptionOperations).call
    @language = "rb"
  end

  def run
    while true  
      print "#@language> "
      line = gets
      break if line.nil?

      if line[0] == ?#
        execute_command line[1..-1].rstrip 
      else
        execute_code read_code(line)
      end
    end
  end
  
  # Reads lines from standard input until a complete or invalid code snippet is entered.
  # Returns ScriptSource that represents an interactive code.
  def read_code first_line
    code = first_line
    while true
      interactive_code = @engine.create_script_source_from_string(code, Scripting::SourceCodeKind.InteractiveCode)
      case interactive_code.get_code_properties
        when Scripting::ScriptCodeParseResult.Complete, Scripting::ScriptCodeParseResult.Invalid:
          return interactive_code
                    
        else
          print "#@language| "
          next_line = gets
          return interactive_code if next_line.nil? or next_line.strip.size == 0 
          code += next_line
      end      
    end
  end

  # Executes given ScriptSource and prints any exceptions that it might raise.
  def execute_code source
    source.execute(@scope)
  rescue Exception => e
    message, name = @exception_service.get_exception_message(e)
    puts "#{name}: #{message}"
  end

  def execute_command command
    case command
      when 'exit': exit
      when 'ls?': display_languages
      else puts "Unknown command '#{command}'" unless switch_language command
    end
  end

  def display_languages
    @engine.runtime.setup.language_setups.each { |ls| puts "#{ls.display_name}: #{ls.names.inspect}" }
  end

  def switch_language name
    has_engine, engine = @engine.runtime.try_get_engine(name)
    @language, @engine = name, engine if has_engine
    has_engine
  end
end

REPL.new.run

As you can see, we’ve encapsulated the code from previous posts into a class called “REPL”. We’ve added a support for meta commands, i. e. commands processed by the REPL itself. Any string starting with “#” chracter is treated as a meta command. The commands recognized are “#exit” and “#ls?”. The former terminates the REPL and the latter displays the available languages. If the command you enter is not one of these two we try to get an engine of that name and change the current language of the REPL. For example, “#py” makes Python the current language, which is indicated by the prompt “py>”.

More of DLR Hosting API

We’ve used a couple of Hosting API concepts that we’ve not explained yet. They are, in order of appearance:

• Engine Services

  ScriptEngine represents a language in Hosting API. We already used it for code execution (ScriptEngine.Execute) and performing dynamic object operations (ScriptEngine.Operations). Apart from these basic functionality an engine can also provide various services that might or might not be language specific. The idea here is that if languages and hosts agree on a new dynamic language service API the languages can implement it and the hosts can consume it without modifications to DLR hosting API. It’s up to each language whether it provides a particular service, so the host should be prepared for the case that some of the languages it hosts might not respond to a service request.

  Services are identified by the CLR type that provides the service API. The service we use in our REPL is ExceptionOperations from Microsoft.Scripting.Hosting namespace. It provides API for handling exceptions, formatting stack traces in a language specific way, etc. We use it in execute_code method to retrieve message and name from an exception raised by the executed script. A default exception operations are provided by DLR if the language doesn’t provide implementation.

• Script Source

  So far we’ve used ScriptEngine.Execute for execution of code stored in a string. ScriptEngine also provides a set of methods prefixed CreateScriptFrom- that create ScriptSource objects representing source code stored in a string, file, stream, an editor buffer, etc. ScriptSource is mainly designed for hosts that need to provide a custom storage for the code, handle encoded code, analyze code, or pre-compile the code so that multiple executions are faster.

  Each ScriptSource instance is also tagged by a value of SourceCodeKind enumeration. This value tells the language parser how to parse the code: whether it should be parsed as an expression, a single statement, multiple statements, like it was a file, or an interactive code snippet. The default value is AutoDetect meaning that the parser should parse any piece of code and decide from its structure what kind of code it is.

  Internally, ScriptEngine.Execute creates a string based ScriptSource with code kind auto-detection and executes it.

  In our REPL, read_code method creates an interactive ScriptSource from a string that user entered. It then queries the ScriptSource for its code properties. ScriptSource asks the parser of the language is it bound to to parse the code and determine whether it is correct and complete, incorrect (has a syntax error), correct but its last token is incomplete (for example, an unterminated string literal), correct but the last statement is incomplete (for example, “1+”). These properties are captured by ScriptCodeParseResult enumeration in Microsoft.Scripting namespace. We use this value in read_code to determine if we should continue reading lines from the input. We continue reading until we get a complete or invalid code.

.NET interop

We have just explained what Hosting APIs we use in the REPL, yet there is still some magic going on in how we call these APIs from Ruby. Let’s look at .NET interop involved here.

• Generic method call

  The first piece of .NET interop magic is used in REPL#initialize method. We have an instance of ScriptEngine in @engine variable and we want to get its ExceptionOperations service. ScriptEngine.GetService<TService> is a generic method. To invoke a generic method you need to provide the generic parameters somehow. The way it works in IronRuby right now is not ideal but one gets the job done nevertheless. Calling Kernel#method on an object gives us a Ruby Method object that represents the (generic) method. IronRuby monkey-patches this class adding Method#of method that takes a list of classes/modules that represent .NET types and returns a new instance of Method class with the method’s generic parameters bound to the given types. Once the generic parameters are bound the method is callable like any other Ruby method (i.e. via Method#call).

• Out parameters

  The call to try_get_engine in switch_language doesn’t seem to be any special at the first glance. However, if you look at the C# implementation of ScriptRuntime.TryGetEngine method, you’ll see that it has two parameters, second of which is an out-parameter:

  bool TryGetEngine(string languageName, out ScriptEngine engine)
  

  When IronRuby encounters a call to such a method it basically takes all the out parameters in the order they appear in the signature and returns them in a CLR array along with the return value. Like if the method’s implementation was as follows:

  object[] TryGetEngine(string languageName) {
    ...
    return new object[] { return_value, engine }
  }
  

  IronRuby is able to splat CLR vector arrays. In fact, any class implementing IList interface can be splatted. Hence the parallel assignment we use in switch_language assigns the return value (a Boolean) to has_engine and the value of the out parameter to engine local variable.

Python interop preview

Finally, we can enter Ruby multi-line snippets and also some Python code!

rb> class C
rb|   def say_hello caller
rb|     puts "#{caller} says hello to Ruby"
rb|   end
rb| end
=> nil
rb> #py
py> import C
py> c = C()
py> c.say_hello("Python")
Python says hello to Ruby

This is a nice little example of Ruby-Python interop that gives you a taste of DLR potential. What’s going on here? First we declared a class C in Ruby with a method say_hello. Then we switched over to Python (using “#py” meta command) and executed some Python code. We imported “C” global variable to the local Python scope. IronRuby maps all constants defined on Object class to DLR global variables. IronPython maps its global variables to the DLR globals as well. Hence IronRuby and IronPython see each other’s variables and can exchange them.

Now that we successfully imported Ruby class object “C” into Python we can call it, right? Wait a second! You can’t call a Ruby class! Well, that’s right, but we are in Python – a call to a class object in Python makes an instance of the class. Python has no new method like Ruby has or new keyword like C# has. When IronPython sees an invocation of a dynamic object that is not its own it sends it an DLR interop message “Invoke”. IronRuby’s class objects don’t respond to this message. After all you can’t call a Ruby class. IronPython is smart and knows that if the object doesn’t respond to “Invoke” message it might respond to “CreateInstance” message. And so it sends the “CreateInstance” message to which the Ruby class responds by sending over a new instance of itself. And after a short chat between IronPython and IronRuby mediated by DLR, we store an instance of class C in Python’s local variable c.

We call say_hello method on the next line. Since we are in Python there are actually two operations involved here: “GetMember” and “Invoke”. Python first asks the object for its member say_hello (by sending “GetMember” message to the object). The Ruby object knows how to get a member – it returns an instance of Method class that represents say_hello method. Python now asks to invoke the returned object and indeed Ruby’s Method knows how to invoke itself.

To run examples below you’ll need to build the latest IronRuby, IronPython and DLR from sources. You can do so using the latest sources from IronRuby GIT repository or DLR CodePlex site:

• Building from IronRuby GIT repo (mapped locally to C:\Git\ironruby):

  C:\Git\ironruby\Merlin\Main > msbuild Languages\Ruby\Ruby.sln
  C:\Git\ironruby\Merlin\Main > msbuild Languages\IronPython\IronPython.sln

  We’ve recently included IronPython projects to the repository. If you already have the repo set up just pull the latest commit. You can also use rake to build IronRuby, we don’t provide rake file for IronPython though.

• Building from DLR CodePlex (downloaded and extracted to C:\CodePlex\DLR_Main):

  C:\Codeplex\DLR_Main > msbuild Codeplex-DLR.sln

Both builds build unsigned debug assemblies and place the binaries into bin\Debug directory.

Available Languages

The DLR runtime can host multiple languages at once. Listing all languages that are available is easy:

# github: languages.rb
# load IronRuby library that provides Ruby programs an easy access to the Hosting API:
require 'IronRuby'

# create a new DLR runtime:
runtime = IronRuby.create_runtime
  
# list all available languages:
runtime.setup.language_setups.each { |ls| puts "#{ls.display_name}: #{ls.names.inspect}" }

The last line enumerates all instances of LanguageSetup class (defined by Hosting API) that are stored in the setup of the runtime. For each such language setup its display name and a list of short names are printed. The short names uniquely identify languages within the runtime. A language can be identified by multiple short names.

C:\Codeplex\DLR_Main\Bin\Debug>ir languages.rb
IronPython 2.6 Alpha: ["IronPython", "Python", "py"]
IronRuby 1.0 Alpha: ["IronRuby", "Ruby", "rb"]
ToyScript: ["ToyScript", "ts"]

Where does this list come from? We have said nothing about IronPython in our script. Does IronRuby know about IronPython? Making IronRuby dependent on all languages it could possibly interact with wouldn’t be a good design. Instead, Hosting API creates and sets up a script runtime according to the current .NET application’s configuration. It is told to do so in IronRuby#create_runtime. The .NET application is ir.exe hence the configuration is loaded from ir.exe.config file that lives next to the ir.exe. Indeed, the list of languages can be found in this file:

<languages>
  <language names="IronPython;Python;py" extensions=".py" displayName="IronPython 2.6 Alpha"
    type="IronPython.Runtime.PythonContext, IronPython, Version=2.6.0.1, Culture=neutral, PublicKeyToken=null" />
  <language names="IronRuby;Ruby;rb" extensions=".rb" displayName="IronRuby 1.0 Alpha"     
    type="IronRuby.Runtime.RubyContext, IronRuby, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />
  <language names="ToyScript;ts" extensions=".ts"    
    type="ToyScript.ToyLanguageContext, ToyScript, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />
</languages>

Whenever you find a DLR language you want to use for scripting your app and don’t have yet you can just add it to the list and you’re all set.

Hosting IronPython in IronRuby

Now that we know how IronPython identifies itself to the runtime, we can load its engine and execute some Python code. Let’s take the simple REPL implementation from the previous post and change it to execute Python code instead of Ruby:

# github: python_repl.rb
require 'IronRuby'
runtime = IronRuby.create_runtime

# ask the runtime for Python engine using one of its simple names:
engine = runtime.get_engine("IronPython")

scope = engine.create_scope

while true  
  print "> "
  code = gets
  break if code.nil?
  begin
    result = engine.execute(code, scope)

    # print the Ruby's view of the result:
    p result

    # print the result as Python would display it:
    puts engine.operations.format(result)
  rescue Exception
    puts $!
  end
end

Let’s play now:

C:\Codeplex\DLR_Main\Bin\Debug>ir python_repl.rb
> def add(a,b): return a + b
nil
None
> add
#<IronPython::Runtime::PythonFunction:0x000005c>
<function add at 0x000000000000002B>
> add(1,1)
2
2
> add((1, 2), (3, 4))
#<IronPython::Runtime::PythonTuple:0x000005e>
(1, 2, 3, 4)
> ^Z

As you can see, each language might have a different view on the same object.

To make the REPL really useful we need to support multi-line statements (i.e. wait with code execution until a complete statement is entered). Let’s keep that and more for the next post.

C:\Program Files\Ruby\bin> irb
irb(main):001:0> class Fixnum
irb(main):002:1>   remove_method :+
irb(main):003:1* end
=> Fixnum
irb(main):004:0> 1
C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/input-method.rb:99:in `gets': undefined method `+' for
3:Fixnum (NoMethodError)
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:132:in `eval_input'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:259:in `signal_status'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:131:in `eval_input'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/ruby-lex.rb:189:in `call'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/ruby-lex.rb:189:in `buf_input'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/ruby-lex.rb:104:in `getc'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/slex.rb:206:in `match_io'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb/slex.rb:76:in `match'
         ... 8 levels...
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:70:in `start'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:69:in `catch'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/lib/ruby/1.8/irb.rb:69:in `start'
        from C:/M1/Merlin/External/Languages/Ruby/ruby-1.8.6/bin/irb:13 
C:\Program Files\Ruby\bin> 

Oops. What happened? We removed method “+” from Fixnum class and, accidentally, irb itself uses that method somewhere in the REPL implementation. The solution is obvious. Run any user code typed in an isolated environment (or “virtual machine”) different from the one that drives the REPL. Let’s take a look how to do this in IronRuby.

Hosting API

The Dynamic Language Runtime (DLR), which IronRuby is built on top of, provides API for hosting dynamic languages in an arbitrary .NET application (see Hosting API specification). You only need to know a few concepts to use this API in a powerful way. The first is script runtime. Each instance of ScriptRuntime class provides an environment for script execution. The runtime hosts scripting language engines. A single runtime can host engines of multiple DLR languages (IronRuby, IronPython, etc.), at most one engine for each language. Each engine is represented by an instance of ScriptEngine class. It holds on a global state that the particular language defines. IronRuby engine, for example, maintains a dictionary of global variables, objects that represent Ruby modules and classes, the list of loaded Ruby files, etc. An application can create any number of ScriptRuntimes and thus any number of IronRuby or IronPython “virtual machines” in a single process.

Hosting API is implemented in C# in Microsoft.Scripting.dll assembly distributed with IronRuby. IronRuby has a rich support for .NET interop, so we could load this assembly and use the Hosting API classes directly from IronRuby scripts. Although it would be simple we made it even easier: IronRuby comes with IronRuby module that provides basic hosting API specialized for Ruby.

Ruby REPL Based on Hosting API

A very simple REPL in Ruby might read as follows.

# load IronRuby library
require 'IronRuby'

# create a new IronRuby engine and a new runtime:
engine = IronRuby.create_engine

while true  
  print "> "
  code = gets
  break if code.nil?
  begin
    p engine.execute(code)
  rescue Exception
    puts $!
  end
end

The REPL script itself runs in a script runtime created by ir.exe host. The code typed in the loop is evaluated in the runtime created by IronRuby#create_engine. The result of ScriptEngine#execute is sent to the calling runtime and printed out via Kernel#p method.

C:\IronRuby\Merlin\Main\Bin\Debug\ir.exe repl.rb
> 1+1
2
> Fixnum.send :remove_method, :+
Fixnum@2
> 1+1
undefined method `+' for 1:Fixnum
> 2.times { |i| puts i }
0
1
2
> ^Z

Note that there are as many class objects for each class as there are runtimes in which the class is used. IronRuby appends “@n” to the name of a class that comes from a different runtime, whose id is n, to differentiate it from the class of the same name in the current runtime. Hence Module#remove_method, which returns the class the method has been removed from, returns “Fixnum@2″.

There are many ways in which we can make our REPL better. The most serious deficiency can be easily demonstrated:

> x = 1
1
> puts x
undefined method `x' for main:Object

Why we got an exception? We defined a local variable x in the first line, so why is it not available in the second line? We are missing some kind of variable dictionary shared among multiple snippet executions. If we used Kernel#eval in Ruby we would pass an instance of Binding to it. The Hosting API uses a similar concept: script scope. I’m going to explain DLR scopes in a separate blog post in detail. Let’s just say for now that IronRuby associates a top-level Ruby binding with each ScriptScope class instance against which it executes code. We need to make a very simple change in our REPL script to use DLR scopes:

# github: repl.rb
require 'IronRuby'
engine = IronRuby.create_engine
scope = engine.create_scope

while true  
  print "> "
  code = gets
  break if code.nil?
  begin
    p engine.execute(code, scope) 
  rescue Exception
    puts $!
  end
end

Local variables work as expected now (actually, the fix in IronRuby source code that makes this work is not yet committed to the public repo, it’s coming soon though):

> x = 1
1
> puts x
1
nil

Another feature our simple REPL lacks is handling of multi-line expressions and statements. Let’s keep that and more for the next blog post.

What distinguishes a flip-flop from a range?

The syntax is the same: expression ‘..’ expression (end-inclusive range) or expression ‘…’ expression (end-exclusive range). A range is considered a flip-flop operator if any of its bound is not an integer literal and if it is used as a condition of the following expressions:

• condition ‘?’ expression ‘:’ expression
• statement ‘if’ condition
• statement ‘unless’ condition
• statement ‘while’ condition
• statement ‘until’ condition
• ‘if’ condition then statements if-tail ‘end’
• ‘unless’ condition then statements else-opt ‘end’
• ‘while’ condition ‘do’ statements ‘end’
• ‘until’ condition ‘do’ statements ‘end’

For example:

irb(main):001:0> 1 if TRUE..FALSE 
=> 1

In addition, parenthesized ranges used in a condition are also flip-flops, e.g.:

irb(main):001:0> 1 if (TRUE..FALSE) 
=> 1

‘(‘ statements ‘)’ is a block expression with a single statement expression TRUE..FALSE. If there are more statements in the block in front of the range it is no longer a flip-flop operator:

irb(main):008:0> 1 if (puts;TRUE..FALSE) 
ArgumentError: bad value for range 

The exception is raised since a range bounds cannot be Booleans.

Any number of nested block expressions works:

irb(main):001:0> 1 if ((((TRUE..FALSE)))) 
=> 1 

And begin-end blocks work as well:

irb(main):032:0> 1 if begin TRUE..FALSE end 
=> 1 

As do any combinations of the two (but only in Ruby 1.9, in Ruby 1.8.6 some combinations don’t work, which I consider a bug):

irb(main):001:0> 1 if (begin (((begin begin TRUE..FALSE end end))) end) 
=> 1 

Also, ranges used in Boolean expressions used as conditions are considered flip-flops:

irb(main):018:0> 1 if t(1)..f(2) and not t(3)..f(4) or t(5)..f(6) 
123456=> 1 
irb(main):018:0> 1 if (t(1)..f(2)) && !(t(3)..f(4)) || (t(5)..f(6)) 
123456=> 1 

where t and f are functions that print the argument and return true and false respectively. The second example also doesn’t work in Ruby 1.8.6, oups.

Let’s summarize what have we just found out: block expressions (parenthesized or begin-end) containing a single range and Boolean expressions propagate the “in-condition” property to their children AST nodes. The “in-condition” property then turns ranges into flip-flop operators. Other nodes don’t propagate it. A method call doesn’t, for example:

irb(main):018:0> 1 if puts(t(1)..f(2)) 
ArgumentError: bad value for range 

Side note: this property is also applied on regular expressions. If a regex is “in-condition” it is compiled as a match against $_ variable:

irb(main):029:0> $_ = 'xz' 
=> "xz" 
irb(main):030:0> 1 if /y/ or ((begin((/x/ and /(z)/))end)) 
=> 1 
irb(main):031:0> $1 
=> "z" 

How does flip-flop work?

We know from the books that flip-flop would probably define some state variable that changes values as the expressions of the operator are evaluated. To figure out how exactly it works I wrote a simple script:

F = false
T = true
x = X = '!'

B = [F,T,x,x,x,T,x,F,F]
E = [x,x,F,F,T,x,T,x,x]
       
def b
  step('b',B)
end

def e
  step('e',E)
end

def step name,value
  r = value[$j]
  puts "#{$j}: #{name} -> #{r.inspect}"
  
  $j += 1
  
  $continue = !r.nil?  
  r == X ? raise : r  
end

$j = 0
$continue = true
while $continue
  if b..e 
    puts "#{$j}: TRUE" 
  else
    puts "#{$j}: FALSE" 
  end
end

In this code we evaluate an end-inclusive flip-flop in a loop. Each time the flip-flop operator is evaluated ‘b’ or ‘e’ method is called, or both methods are called. Global variable $j is an index into B and E arrays defined at the top. The arrays define a sequence of values the methods b and e should return. If x value (‘!’) is to be returned an exception is thrown. This way we can track what the automaton behind the flip-flop operator does. Using the output of the script we can easily deduce how the automaton could look like:

0: b -> false
1: FALSE
1: b -> true
2: TRUE
2: e -> false
3: TRUE
3: e -> false
4: TRUE
4: e -> true
5: TRUE
5: b -> true
6: TRUE
6: e -> true
7: TRUE
7: b -> false
8: FALSE
8: b -> false
9: FALSE
9: b -> nil
10: FALSE.

It has four states: BEFORE, INSIDE, AFTER and a lambda state. In each state b or e expression is evaluated and the automaton transitions to another state based upon the result: true (T) or false (F). If it transitions to INSIDE or AFTER state the flip-flop operator returns true. If it transitions to BEFORE state the operator returns false. If it transitions to the lambda state the operator evaluates b and does one more transition.

The previous automaton works for an end-inclusive flip-flop operator. The one bellow does for an end-exclusive operator – the lambda state is gone:

What code does IronRuby emit?

We emit the following code for end-inclusive flip-flop operator {begin}..{end}:

if state || IsTrue({begin})
  state = IsFalse({end})
  true
else  
  false
end  

The state variable is a Boolean and it’s true iff the automaton is in INSIDE or lambda state. In both states this means evaluate {end} expression, transition to the state given by negated result and return true. If the state variable is false we are either in AFTER or in BEFORE state. In any case {begin} is evaluated. If it returns false we go to/stay in BEFORE state and the result of the flip-flop operator is false. Otherwise we go to the lambda state and evaluate end.

Similarly, the end-exclusive flip-flop operator {begin}…{end} is implemented as

if state
  state = IsFalse({end}) 
  true
else
  state = IsTrue({begin})
end

The state variable is allocated in the inner-most method scope. A flip-flop operator used in a block preserves the state across multiple calls to the block:

def y *a; yield *a; end

def test
  $p = proc { |b,e|
    puts b..e ? TRUE : FALSE
  }
  
  y false, &$p  
  y true, true, &$p
  y false, &$p
  y true, false, &$p
end

test

Output:

false 
true 
false 
true 

The method that transforms IronRuby’s RangeExpression node into DLR AST in the case the range is a flip-flop operator looks like:

private MSA.Expression/*!*/ TransformReadCondition(AstGenerator/*!*/ gen) {
    // Define state variable in the inner most method scope.
    var stateVariable = gen.CurrentMethod.Builder.DefineHiddenVariable("#in_range", typeof(bool));

    var begin = Ast.Box(_begin.TransformRead(gen));
    var end = Ast.Box(_end.TransformRead(gen));

    if (_isExclusive) {
        return Ast.Condition(
            stateVariable,
            Ast.Comma(Ast.Assign(stateVariable, Methods.IsFalse.OpCall(end)), Ast.True()),
            Ast.Assign(stateVariable, Methods.IsTrue.OpCall(begin))
        );  
    } else {
        return Ast.Condition(
            Ast.OrElse(stateVariable, Methods.IsTrue.OpCall(begin)),
            Ast.Comma(Ast.Assign(stateVariable, Methods.IsFalse.OpCall(end)), Ast.True()),
            Ast.False()
        );
                          
    }
}

This code can be found in Ruby\Compiler\Ast\Expressions\RangeExpression.cs. Propagation of “in-condition” property to AST nodes is performed by ToCondition virtual method overridden by RangeExpression, RegularExpression, AndExpression, OrExpression and BodyExpression and called from parser (Ruby\Compiler\Parser\Parser.y).