Thursday, September 25, 2014

How to: Tokenize a string with Typescript and Regular Expressions


Source Code: https://github.com/WCoetser/RandomUsefulCode/tree/master/TypeScriptTokenizer

Sometimes you have a string, and you want to identify sub-strings in this string by matching them with regular expressions. I wrote this small program to do that: (see full source code here)

Step 1: Specify the regular expressions and sub-strings

(Note: Tokenizer is not a build-in class, I wrote it. See step 2. See above link for the rest.)

var tokenizer = new Tokenizer();      
tokenizer.addTokenDefinition(TokenType.Variable, /\:\w+/);
tokenizer.addTokenDefinition(TokenType.Identifier, /[a-zA-Z_]\w*/);
tokenizer.addTokenDefinition(TokenType.String, /"[^"]*"/);
tokenizer.addTokenDefinition(TokenType.Number, /[+-]?[0-9]+[\.]?[0-9]*/);
tokenizer.addTokenDefinition(TokenType.Whitespace, /[\s]+/);

Step 2: Tokenize

class Tokenizer {

  private definitions: Array;

  constructor() {
    this.definitions = new Array();
  }

  addTokenDefinition(type: TokenType, regex: RegExp) {
    var tokenDef = new TokenDefinition();
    tokenDef.type = type;
    tokenDef.regex = regex;
    this.definitions.push(tokenDef);
  }

  tokenize(strIn: string): TokenizationResult {
    var retVal = new TokenizationResult();
    var sourceString = strIn;
    if (!strIn || (strIn.match(/^[\s]*$/) != null)) {
      retVal.succeed = false;
      retVal.message = "Error: Empty input given."
      return retVal;
    }
    var nextParsePosition = 0;
    var matchedType = TokenType.None;
    retVal.result = new Array();
    do {
      matchedType = TokenType.None;
      this.definitions.forEach(def => {
        var match = def.regex.exec(sourceString);
        if (match != null && match.index == 0 && match.length >= 1) {
          // Use the string indexOf member to skip the unwanted characters
          var token = new Token(strIn, nextParsePosition, match[0].length, def.type);
          retVal.result.push(token);
          // Note: It is neccesary to skip by the amount adding in non-matched characters from the first group.
          nextParsePosition += match[0].length;          
          sourceString = sourceString.substr(match[0].length);
          // Continue to next token 
          matchedType = def.type;
          return;
        }
      });
    }
    while (nextParsePosition < strIn.length && matchedType != TokenType.None);
    retVal.succeed = nextParsePosition == strIn.length;
    if (!retVal.succeed) retVal.message = "Unexpected characters at position " + nextParsePosition;    
    return retVal;
  }
}


Points of interest

Wednesday, September 24, 2014

How to: Setup a Github repository and do an initial upload on Windows

This guide is a quick how-to for creating a new Github repository, and uploading new files via the command prompt. The Windows interface was too confusing for me, so I ran back to the command line :-) Note that this is a guide assumes that you are new to Github and does everything in the master branch. Therefore I do not do forking etc.

Step 1: Create a Github account if you don't have one

This can be done by visiting https://github.com/ and signing up. For this guide you will need a Github username and password.

Step 2: Create a repository

Create a repository by clicking on the button in the screenshot:

The Github repository creation button
Follow the instructions on the screen until you have a repository.

Step 3: Download Github for Windows

Get it here: https://windows.github.com/

Step 4: Open Github for Windows, and browse to the Github shell


Where to find the Github shell in Github for Windows
This will open Powershell with the Github extentions: (it will be blank at this point)

The Github for Windows Powershell extensions in action
Step 5: Initialize the repository locally

Change directory to the parent of the directory where you want to host your code. The directory must not exist yet:

cd D:\Github

Now get the new repository that you created in step 2: First copy the repository HTTPS URL: You can copy and paste it from the website in the location in this screenshot.

Where to find your repository HTTPS URL
Now the repository can be initialized by cloning it to your hard-drive. Go back to Powershell and run this command:

D:\Github> git clone https://github.com/WCoetser/RandomUsefulCode.git

Note that https://github.com/WCoetser/RandomUsefulCode.git is what I copied and pasted. This command will create a subdirectory, in this case D:\Github\RandomUsefulCode

Step 6: Create your content and copy it to the local directory

The local directory here is D:\Github\RandomUsefulCode. If you have existing old content in the local directory, overwrite it with the new content.

Step 7: Upload your code to Github

First change to the directory containing the code:

D:\Github> cd .\RandomUsefulCode

Now add the new files. You can always run this step, just to be sure nothing goes missing:

git add -A

Now commit all changes:

git commit -a -m "Upload tokenizer version 1"

The part in quotes is the description of the changes. Please note that these changes have only been "locally committed". Therefore it is not uploaded to Github on the Internet yet. In order to upload it, you will need to copy the HTTPS URL from step 5 again. With the URL, run this command:

git push https://github.com/WCoetser/RandomUsefulCode.git master

This will ask you your Github username and password from step 1.

Step 8: Check your upload

You can check that your files have been uploaded by browsing to your repository after you logged in. The files usually appears immediately.


Sunday, August 24, 2014

The potential of Enterprise Service Busses: First Impressions

In the past month or so I had my first real exposure to an Enterprise Service Bus (ESB), and I decided to write down my experiences in order to get my thoughts lined up.

First of all we may ask ourselves what an ESB is. From what I could tell, it is a piece of software that is used to connect various systems that have been developed in isolation in such a way that they form a product that is greater than the sum of their parts. So, for example, it would be possible to connect an accounting system to a stock tracking system in such a way that new stock is automatically ordered when stock runs low.

Neither the stock tracking nor the accounting system needs to know about each other, they must just implement some commonly acknowledged standards that allows them to communicate with external systems.

This is what sets ESB based integration apart from normal systems integration. In normal systems integration, the systems integrating with each other must know each other and in most cases are designed for each other, or at least one system is designed for the other.

It seems that there is a lot of potential for Enterprise Service Busses to move into a space that was previously occupied by traditional client/server technology. It is, for example, possible to hit a REST interface exposed in an ESB from a website. Therefore the ESB takes over the traditional role of the server in the client/server architecture. It should be possible to create entire systems by just writing stored procedures on a database for the back-end and HTML/JavaScript/CSS for the front-end. These systems will have not necessarily have Java or C# components.