Barrie Treloar

Friday, December 21, 2012

Lexer and Parsers tests for XText

See for the original blog entry that this is a reply to Basic lexer and parser tests for your Xtext grammar blog by itemis

The following code is updated to use XText 2
It is not abstract, so that you can instantiate it to debug tokens from elsewhere.
All methods are public so you can use it from elsewhere.

package test;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import org.antlr.runtime.ANTLRStringStream;
import org.antlr.runtime.CharStream;
import org.antlr.runtime.Token;
import org.eclipse.xtext.GrammarUtil;
import org.eclipse.xtext.ParserRule;
import org.eclipse.xtext.junit4.AbstractXtextTests;
import org.eclipse.xtext.parser.IParseResult;
import org.eclipse.xtext.parser.antlr.ITokenDefProvider;
import org.eclipse.xtext.parser.antlr.Lexer;
import org.eclipse.xtext.parser.antlr.XtextTokenStream;

import com.github.usrprops_xtext.parser.antlr.UsrpropsDslParser;

public class LexerAndParserTest extends AbstractXtextTests {

    private Lexer lexer;

    private ITokenDefProvider tokenDefProvider;

    private UsrpropsDslParser parser;

    private boolean debugTokenStream = false;

    public boolean isDebugTokenStream() {
        return debugTokenStream;

    public void setDebugTokenStream(boolean debugTokenStream) {
        this.debugTokenStream = debugTokenStream;

     * return the list of tokens created by the lexer from the given input
    public List<Token> getTokens(String input) {
        CharStream stream = new ANTLRStringStream(input);
        XtextTokenStream tokenStream = new XtextTokenStream(lexer, tokenDefProvider);
        List<Token> tokens = tokenStream.getTokens();
        return tokens;

     * return the name of the terminal rule for a given token
    public String getTokenType(Token token) {
        return tokenDefProvider.getTokenDefMap().get(token.getType());

     * @param tokens
     *            the tokens to debug to System.out
    public void debugTokens(List<Token> tokens) {
        for (int i = 0; i < tokens.size(); i++) {
            Token token = tokens.get(i);
            System.out.println("Token type=" + getTokenType(token) + " text=" + token.getText());

     * check whether an input is chopped into a list of expected token types
    public void checkTokenisation(String input, String... expectedTokenTypes) {
        List<Token> tokens = getTokens(input);
        List<String> actualTokenTypes = new ArrayList<String>(tokens.size());
        for (int i = 0; i < tokens.size(); i++) {
            Token token = tokens.get(i);
            if (debugTokenStream) {
        assertEquals(input, Arrays.asList(expectedTokenTypes), actualTokenTypes);

     * check that an input is not tokenised using a particular terminal rule
    public void failTokenisation(String input, String unExpectedTokenType) {
        List<Token> tokens = getTokens(input);
        assertEquals(input, 1, tokens.size());
        Token token = tokens.get(0);
        assertNotSame(input, unExpectedTokenType, getTokenType(token));

     * return the parse result for an input given a specific entry rule of the
     * grammar
    public IParseResult getParseResult(String input, String entryRule) {
        ParserRule rule = (ParserRule) GrammarUtil.findRuleForName(getGrammarAccess().getGrammar(), entryRule);
        return parser.parse(rule, new StringReader(input));

     * check that the input can be successfully parsed given a specific entry
     * rule of the grammar
    public void checkParsing(String input, String entryRule) {
        IParseResult la = getParseResult(input, entryRule);
        assertFalse(input, la.hasSyntaxErrors());

     * check that the input cannot be successfully parsed given a specific entry
     * rule of the grammar
    public void failParsing(String input, String entryRule) {
        IParseResult la = getParseResult(input, entryRule);
        assertFalse(input, la.hasSyntaxErrors());

     * @param ruleName
     *            name of the terminal rule
     * @return The name of terminal rule will be capitalised and "RULE_" will be
     *         appended to the front of the ruleName
    public static String rule(String ruleName) {
        return new StringBuilder("RULE_").append(ruleName.toUpperCase()).toString();

     * Return the rule name for a keyword, which is the keyword enclosed in
     * single quotes
     * @param keyword
     *            the keyword
     * @return the rule name for the keyword
    public static String keyword(String keyword) {
        return new StringBuilder("'").append(keyword).append("'").toString();

     * check that input is treated as a keyword by the grammar
    public void checkKeyword(String input) {
        checkTokenisation(input, keyword(input));

     * check that input is not treated as a keyword by the grammar
    public void failKeyword(String keyword) {
        List<Token> tokens = getTokens(keyword);
        assertEquals(keyword, 1, tokens.size());
        String type = getTokenType(tokens.get(0));
        assertFalse(keyword, type.charAt(0) == '\'');

Friday, March 31, 2006

So long and thanks for all the hot dogs (apoligies to Douglas Adams)

Well that brings me to the end of my New York jaunt.

At midday I will be heading to JFK airport to start my trek home.



MoMA was a little disappointing.
There was some cool stuff there like picaso and a few others.
But some of the stsuff is just junk, either plain coloured canvases or really amatuerish looking drawings/paintings that my nieces have done better.

Stopped by chop't again since it was so close and had a tuna salad. Shouldn't eat raw red onion, smelly breath!


Go see Lion King

The only tickets to Lion King available were the expensive ones at $110 each. Ouch.
I bought one for tonight (wednesday).

Visited the USS Intrepid which is awesome. Not enough of the ship is open to touring but its enough to wander around in and look at all the exhibits. Afterwards headed to the rockefeller centre where Nintendo World is.

I love Nintendo World, it is Pokemon heaven. So much stuff!!!!
Marion isn't going to like all the things I'm going to buy when I get home.
I need a Nintendo GameCube with Pokemon Stadium (It has story mode and it's the one I have been looking for for ages but couldn't remember), plus Pokemon XD, and maybe since I need a GameCube some super mario and/or mario party games. *shiver* I can't buy these in america because they are region locked ;(

Time for dinner before Lion King, I stopped at Tony's Italian restaurant and had Spaghetti and Meatballs. Quite yummy but way too large, most of their dinner meals are meant to be platters you can share with 2 or 3. Nice place. The person sitting next to me at the bar is an ex New York cop and has arranged for me to get a tour of operations thursday night at 9pm. Should be good.


Wednesday, March 29, 2006

Long lines suxor

I shoulda listened more closerly to the concierge when given advice, or at least not be so tired and to think straight.

I paid $37 to get the hop-on-hop-off bus but got off at battery park to see the statue of liberty at 10:30 in the morning and only got back to the mainland at 2:30. It was over 2 hours before I even got into the pedestal and if I hadn't dawdled in the museum part would have been over in 5 minutes.

After that annoying waste of time, I wandered up to wall street (how tiny is that!) and waved at the bull statue as I went past. Then on to ground zero and the church nearby.

At 4:45 I get the hop-on-hop-off bus again (and hell is it cold on the top deck) only to be turfed off at Times Square because this was the last bus. I was hoping to get dropped off at Macy's so I could get warm in the hotel. Instead I wandered to Virgil's BBQ restaurant for a rib + chicken meal. It was worth trying but not worth eating. Anyhoo back to the hotel to warm up. It's not so far from Times Square after all.

One bright note is that the bus went past the Rockefeller building and it did look like I should go have a look. So still to do:

USS Intreprid
Lion King.

The tour driver pointed out Tiffany's but I couldn't see it. Marion didn't really want anything from there anyway. :)


Tuesday, March 28, 2006

Monday's Wanderings

Today was a nice weather day. The rest of the week isn't meant to be as good.
I hope it stays pretty fine so I can get more outdoor stuff done.

Macy's is next door to the hotel so I had a look. It's pretty much like Myer, a couple of interesting things but not at the prices they are asking. (and I don't have any room left in the luggage anyway). Zoomed through the floors quickly so as to not waste too much time and then headed to Penn Station.

I'm getting the hang of the subway thing. Other people are asking me for directions! They must be crazy. I don't know where I'm going. Anyway, took the subway and changed over to the S shuttle between 42nd and Central. You can see in the photos how run down these subways are. No need to build a set for post-apocolyptic movies. Central was quite nice, pity the photos didn't come out too well.

After leaving Central I went looking for the Borders at 461 Park Ave and Chop't on 57 between Madisson and Park. I found Borders first. I love Borders! They had my Dragon Ball Z comics. ALL of em. I bought up!!! USD$7.95 and no postage, bargain. It'd cost me AUD$17.95 back home. Wandered around the sci-fi/fantasy books but couldn't see anything worth buying which is lucky because my Eclipse bag is going to be stuffed full of books as it is.

After Borders I went looking for lunch, I finally found Chop't. I don't know where I read or heard about these people. All I can remember is that someone said that this place should be in Australia and I have to agree. Check out the line up at 1:30pm!!! The line moves pretty quick. They have 3 stations with 3 people at each station. Someone to put your ingredients together and another to chop and put into your bowl. It's not cheap at $9.25 for a salad bowl but it's a huge bowl, I was stuffed afterwards. I chose the Gotham; iceberg lettuce, grilled chicken, black forest ham, white cheddar, tomatoes, beets, crumbled egg, smoked bacon with a smokey bacon russion dressing. If you add up those ingredients from the custom list it would cost $11+.
Chop't feels like one of those franchises that would just take off like subway has.

I was planning on either MoMA or a cycle around Central Park. It was such a nice day I decide Central Park, but I was too stuff from Chop't to go cycling. I bumped into the Zoo while at the park so went it. It is quite tiny but clean and the animals are fun. Wollman Rink was nearby so saw that too, as well as the chess house. The signs in the park say no bicycles so maybe I can only ride round the running track? That was too far away to check out so I went walking back down 6th Ave to see the Rockefeller Center on the way to see about Broadway tickets.

There were quite a few people out the front of Radio City but the Rockefeller Center didn't look appealling to me from the outside so kept walking past to the Broadway ticket booth. I was given a guide which showed what tickets they usually sell there. Of course all the cool shows I would go and see don't go for cheap prices (The Lion King, Monty Python's Spamalot) although looking more closely Rent and Producers have a chance. Rent was going 35% cheaper today.
I think I'd rather see Lion King though. I'll wander to the box office and see what I can purchase they run from Wed-Sat at 8pm, so I will only have Wed and Thursday to choose from.

Had a look at Spoiled Brat on 49 between 8/9 ave, it's a pet store but has mostly dog stuff, so the kittens have lucked out on getting a toy from overseas.

By this time my feet are sore from so much walking, time for another Subway jaunt.
I had been wondering why people say New York is crowded. Admittedly I hadn't been moving around during peak times yet. At 4:30ish the Subway was quite full, enough room to squeeze on but I can see what they mean.

Still need to find a post office to get some stamps so I can send post cards.
The vague directions I was given to the post office (a block over on 8th) didn't help me to find it. On the way back I bought some more groceries and a can of vegie and noodle soup for dinner.

Left on the itineray:
* MoMA
* USS Intrepid
* Central Park (more of it???)
* Broadway Show
* Wall Street
* Statue of Liberty

I might take a tour bus tomorrow on the Downtown loop which will cover some of those.

Now for some sleep.


Monday, March 27, 2006

Sunday 27th March

Is 11am early?

Anyway, armed with all my trusty tourist handouts from Tina at concierge I caught the subway blue line at Penn Station to Guggenheim. The subway is not that confusing!!!
I managed to work out I needed the AC uptown blue line (and not the AE blue to queens).
But I wondered what the difference between local and express was. Pity the express arrived first.

The express meant that from 59 St it was express to 116 St and I wanted to get off smack in the middle. After hoping off and getting on the local blue line back downtown I jumped off at 86 St and cut across the road through Central Park.

A short walk later I arrived at the Guggenheim to see the David Smith sculpture collection.
Some of his stuff is cool, I prefer the Steel sculptures to the stainless steel ones. I especially liked the large ones that were very minimalistic with wheels on them. Of course I can't find any links to the sculptures I liked. You can buy the book with the sculptures in them, I had a quick browse as I thought Kali should buy it for her Dad (the work reminded me of his exhibition) but the book doesn't do the work justice, you gotta see this stuff in the flesh.

Part of the permanent collection includes some work by Picasso which was nice to see.

Three hours later I've finished the tour and head outside and grabbed a cab to Barnes and Nobles which has 4 floors of books and cds. The prices of stuff is pretty much the same as home and definately didn't have more Dragon Ball Z comics!!! I'll have to try Borders later.

Wandered down broadway past the ticket booth (nothing cheap for lion king, rent, spamalot, producers or stomp) will check again each day.

Past Toys R Us, I love that shop. I want a Hover Copter but I think I will need one with a proper Aussie power supply. And the Star Wars Light Saber Battle Game.

Back towards the hotel and grabbed two slices of pizza. They were ok. Won't bother again.


Saturday 26th March

I didn't wake up until 1pm. Long hours at EclipseCon and staying up late to ring Marion are taking it's toll.

The afternoon was spent talking to the internet help desk and hanging around in the hopes they would get it fixed. At 6pm I gave up and talked to the concierge about where to head for dinner.
Tina suggested Seven (located at Map) where I had

Grilled Roasted Chicken
olive oil crushed Yukon gold potatoes,
asparagus spears, natural jus

which is was tasty smokey chicken. The sonoma county cab sav was good too.

I then went for a wander down 7th ave to Carmine St but google maps isn't too good sticking to 7th Ave :)

Concierge said the Empire State Building was open until 11pm which gave me 1 hour to get there and wander around. I wish I had taken picture of the lovely red carpet with fancy red ropes. It zigs-and-zags for ages!!! Thank god there were no lines when I went through. There was enough of a wait further along anyway. More photos on Flickr



Photos are available at Flickr


Arrived in New York City Friday 24th March

After leaving EclipseCon at 7:30am I arrived dead tired at Newark airport 7pm New York Time.

Found the bus station at the airport and travelled to the Ports Authority Bus Termin in the heart of New York. The bus driver wasn't too helpful with directions, other than pointing the way to 7th Ave. Since I was on 8th Ave I figured I didn't have far to go.

The Ports Authority is on W 42nd and Affinia Hotel is between W 30 and W 31.
Check this map for an indication of how far I walked wheeling my luggage behind me.

They have high speed internet access in the room but at the rude price of $50 per week.
It was free at the Holiday Inn Santa Clara. What's worse was it stopped working at 1:30am when I wanted to ring Marion. They didn't get it fixed until Sunday morning 10pm.