Expressions notations

What kind of notations for expression syntax do you know? Basincs are:

• INFIX: 5 * 2 + 3 = 13 (implicit precedence)
• POSTFIX: 5 2 * 3 + (stack based, explicit precedence)
• PREFIX: + 3 * 5 2 (reversed postfix)
• NESTED: (+ (* 5 2) 3) (lisp/s-expressions)
• FUNCTIONAL: +( *(5 2) 3) (m-expression)
• MATCHFIX: (* (+ 5 2 +) 3 *) (remembed "end/end if/fi"?)

Another one?

Some of them are very easy for implementation and parsing is trivial (prefix/postfix/nested/matchfix), but more human readable is infix which leads to creation of parsing theory.

Original

Simplest config file parser in the world!

Nothing then split/strip/filter "combinators":

cfg = '''
    a   = aa aaa = aaaa
b   = bbb
        c=cccccccccc


'''
split_pair = lambda s: s.split('=', 1)
strip_pair = lambda p: map(str.strip, p)
parse_rc = lambda s: map(strip_pair, map(split_pair, filter(None, s.splitlines())))
print dict(parse_rc(cfg))  # print lik in Python 2x

Roadmap to NanoLP 1.0

1.0 is planned to be beta version. So plans are:

• several new features(for collabortive usage, flex text transformation, etc.)
• tests improvement
• fixing tests on Linux (problem with paths)
• fixing lookup of installed files on Linux (ploblem with paths)
• README files for every test
• presentation and Quick References
• other BUGFIX

And possible another documentation materials...

NanoLP 1.0i released

Special Web edition: now LP code base can be published on the Web as online documentation and at the same time - as library (for including/importing into your LP programs).

Added new functions, new parser, total code refactoring, added events handling mechanism (with chaining support), URL support in filenames, Web-publishing and more...

New Literate Programming Tool - TCLP 1.0!

I done new LP tool: very small (about 700-800Kb!) and flexible!
Main features are:

1. Document has 'document header' with fields: TITLE, PROJECT, VERSION, DESCRIPTION, AUTHOR, ADDRESS, TAGS, it shown on the top of document.
2. User can define values of this fields (and any other field) and can use defined fields in the document body (even in code chunks). There are special fields: APPNAME, APPVERSION, TIMESTAMP, DATE, TIME.
3. Document has auto-generated TOC (table-of-contents), IT (index-table), ERT (external-references-table). TOC is placed in the start of the document. IT is placed at the end of document. ERT is placed last.
4. TOC is generated automatically for all headers, defined with 'head' command.
5. IT is generated automatically for all code chunks and includes file names of output chunks and lines numbers where to find them.
6. ERT is generated automatically for all external references in the document, defined with 'ref' command and includes back-references to it's usage in the document body.
7. Markup commands includes commands for bold, underlined, italic and monotyped text. Top/bottom indexes are supported also. Markups can be nested.
8. There is special file with abbreviations to be used in section names and acronyms also.
9. User can use acronym in the text from abbreviations file or explicitly with embedded expansion.
10. User can create "vocabularies" - tables of definitions - from abbreviations file or embedded, defined explicitly.
11. There are named counters (auto-incremented) to use in captions and so on.
12. Code chunks are special formatted fragments of source code with lines numbering, optional syntax highlighting (only keywords). Lines can be numbered in continuously manner (with the same counter names) or in independent. User can reference to code lines by chunk name (section) and line number or with defining NAME of code line and usage of this name for referencing.
13. Tables are supported: with attributes (table/row/cell), spanning, nesting.
14. Lists (ordered/unordered) are also supported: with attributes; they can be nested also.
15. User can insert image with 'embed' command. Any fragment of document can be "framed" with 'frame' command. It creates frame with caption (on top/on bottom), name and links (for referencing).
16. There are 2 kinds of citations: inlined and blocked. Citation can have linked URL (shown as "More...")
17. User can insert in document text any previous defined field values, but also variables from environment. Special environment variables is the USER - this is the name of user running tool.
18. There is a special command 'wrap' for wrapping text with any word or quotation symbols, braces, etc.
19. There is a command 'ent' to translate special symbols to HTML entities.
20. There are references:
    • to external URL
    • to header
    • to line of code chunk/named line of code chunk
    • to code chunk
    • to frames
    • to any place of document where user defines own link
21. There are 2 special commands: 'atstart', 'atend'. Mainly used is 'atend' for code generation from code chunks. User can define there what code chunks to be written to output source files (by sections, by glob-pattern of sections). Also is possible to write any free text to output file.
22. Most of created HTML tags has CSS-classes. For all available CSS classes, see 'tclp.css' file. Used CSS file can be defined with command line option.
23. User can add syntax keywords for any other language (for highlighting).

You can try it from Google Projects: http://code.google.com/p/tclp/

Automatic generation of proc/methods help in Tcl

Often you write not only internal code but procs for user's usage (in console and so on). It's difficult to write documentation of procedures in different file, so let's use doc-strings like Lisp, Python and so on.
It's not Java, so you don't need special tool :)
Considering that out procedures/IncrTcl methods has comments like this:

proc f {args} {
## Doc-string for this method.
# Syntax: f ...
##
... CODE ...
}

OR even:

proc f {args} {
## Doc-string for this method.
# Syntax: f ...
# Other help ##
... CODE ...
}

Well, doc-string is at proc body start, and begins with '##' and ends with '##' (at the same line or next), other comments are treated as usual comments and are ignored for doc-string mechanism. Here is example how to parse suck doc-strings:

package provide mydoc 1.1

package require Itcl

namespace eval mydoc {
    namespace export doc
}

set _NOHELPMSG "No help."

proc ::mydoc::_docstring {body} {
    set body [string trim $body]
    set docstring ""
    # without 1st '^' will match any FIRST docstring block even after
    # commands!
    if {[regexp {^##\s*([^\n]+\n?)+(##)} $body docstring]} {
        set docstring [regsub -all {\s*#\s?} $docstring \n]
        set docstring [string trim $docstring]
        return $docstring
    }
}

proc ::mydoc::doc args {
    ## Help on command: procedure or class method. Call:
    #   doc some_object some_method
    #   doc some_class some_method
    #   doc some_proc ##
    global _NOHELPMSG
    set found ""
    switch -- [llength $args] {
        1 {
            # args: proc
            set name [lindex $args 0]
            set arguments [info args $name]
            set body [info body $name]
            set found [_docstring $body]
        }
        2 {
            # FIXME not optimal!
            # args: object|class method
            lassign $args cls_obj meth
            set objs [itcl::find objects]
            # cls_obj may be object OR class. What is it?
            if {-1 != [lsearch -regexp $objs :*$cls_obj]} {
                # this is the object
                set arguments [$cls_obj info args $meth]
                set body [$cls_obj info body $meth]
                set found [_docstring $body]
            } else {
                # this is the class
                set arguments [namespace eval ::$cls_obj info args $meth]
                set body [namespace eval ::$cls_obj info body $meth]
                set found [_docstring $body]
            }
        }
        default { error "wrong args: proc | object method | class method" }
    }
    if {$found eq ""} {
        return $_NOHELPMSG
    } else {
        return $found
    }
}

Usage is simple, you can call:

doc object method
doc class method
doc usual-procedure

Class - is the IncrTcl class

PEG grammars in Tcl - simple example

This is the very simple example how to create PEG grammar and parse some text with it in Tcllib (I used Tcllib 1.4).

package require starkit
starkit::startup
package require pt::peg::from::peg
package require pt::peg::container
package require pt::peg::interp

set PEG {
PEG myapp (Exp)
    IfKw       <- 'i' 'f'                               ;
    ThenKw     <- 't' 'h' 'e' 'n'                       ;
    ElseKw     <- 'e' 'l' 's' 'e'                       ;
    OSp        <- ('\t' / '\n' / ' ')?                  ;
    Sp         <- ('\t' / '\n' / ' ')+                  ;
    Block      <- (Exp / 't') &';'                      ;
    Cond       <- 'c' / 'o'                             ;
    If         <- IfKw Sp Cond OSp ':' OSp Block        ;
    Exp        <- If                                    ;
END;
}

set CPEG [pt::peg::from::peg convert $PEG]
pt::peg::container cont
cont deserialize = $CPEG
set src "if c: if o:t;"
pt::peg::interp myint
myint use cont
puts [myint parset $src]

(I use startkit...)
This grammar is very simple (and non-realistic): it defines rules to parse nested 'if'-expression like this:

if c: if o: t;

OR

if CONDITION: if CONDITION: BLOCK;

where CONDITION is 'c' or 'o', BLOCK is another EXP (expression) or 't' followed by ';'; when blocks are nested, only one ';' is needed. Example has bugs (see Sp, OSp - Optional SPaces) but shows base tricks with PEG, for ex., PEG definition is loaded from variable, not imported from file, so you need for your App. not all Tcllib packages :) After running, you get:

c:\>tclkitsh.exe main.tcl
Exp 0 11 {If 0 11 {IfKw 0 1} {Sp 2 2} {Cond 3 3} {OSp 4 3} {OSp 5 5} {Block 6 11 {Exp 6 11 {If 6 11 {IfKw 6 7} {Sp 8 8}
{Cond 9 9} {OSp 10 9} {OSp 11 10} {Block 11 11}}}}}

Keyword (optional) procedure arguments in Tcl

They are very usable in many language. For example in Python:

def f(a, b=0, **kw):
  print kw

f(1, 2, x=10, y=20)

kw is the dictionary and is accessible in usual way (item existent checking, getting and so on). Argument b can be skipped and is optional too.

For Tcl it's easy to write procedure for processing keyword arguments (like getopt):

proc getargs {argsName kwArgsName arguments} {
# like getopt for proc/method, ex:
# 1) argsName may be ""|{} - to avoid positional args saving
# 2) if no value of keyword argument, then it's value will be "Y"

    upvar 1 $kwArgsName kwargs
    set posargs {}

    proc isKw opt {
        return [expr {"-" eq [string index $opt 0]}]
    }
    proc setPosArg opt {
        upvar 1 posargs posargs
        lappend posargs $opt
    }
    proc setKwArg {name val} {
        set ret "@NOMORE"; # used or not $val, it's possible positional opt
        if {$name ne ""} {
            upvar 1 kwargs kwargs
            set defaultValue [lindex [array get kwargs $name] end]
            if {$defaultValue in {"Y" "N"}} {
                if {$val eq ""} {
                    set val "Y"
                } elseif {$val ni {"Y" "N"}} {
                    set ret $val
                    set val "Y"
                }
            }
            set kwargs($name) $val
        }
        return $ret
    }

    set kw ""
    set state OPT; # OPT|VAL|END
    lappend arguments "@EOO"
    foreach opt $arguments {
        switch -- $state {
            OPT {
                if {$opt eq "@EOO"} {
                    set state END
                } elseif {[isKw $opt]} {
                    set kw $opt
                    set state VAL
                } else {
                    setPosArg $opt
                }
            }

            VAL {
                if {$opt eq "@EOO"} {
                    setKwArg $kw ""
                    set state END
                } elseif {[isKw $opt]} {
                    setKwArg $kw ""
                    set kw $opt
                } else {
                    set maybePos [setKwArg $kw $opt]
                    if {$maybePos ne "@NOMORE"} { setPosArg $maybePos }
                    set state OPT
                }
            }

            END { break; }
        }
    }

    if {$argsName ne ""} {
        upvar 1 $argsName _posargs
        set _posargs $posargs
    }
}

Here is the example how to use:

set args {}; # positional args
array set kw {-a 1 -b 2}; # default kw-args
set inputArgs {-a 20 -c 30 zzz -q -v xxx yyy}
getargs args kw $inputArgs
puts "Args:\n $args"
puts "KW Args:"
foreach {k v} [array get kw] {
    puts " $k = $v"
}

If positional arguments are not needed, use "" or {} instead of its' variable name. If some keyword options has no value (it's flag only!), set it's default value ("Y" or "N") to kwArgsName (kw array is example), so getargs will not interpret the next word as option value. But it's possible to has value if it is "Y" or "N": someCommand -someFlag or someCommand -someFlag N

Parsing of paths

Paths are often used in software environments: in file systems, in configuration files, in different IDs and so on. Here are two Python functions, used for parsing of them, they generate tracks from paths (function dothier()) and map based trees (function dottree()). Here you are:

def dothier(seq, sep='.'):
    """Generate path from dot-strings in order for creation
    tree: first - parents then children:
    >>> for x in dothier(["x.y.z", "a"]): print x, x.leaf
    a True
    x False
    x.y False
    x.y.z True

    Every returned string (unicode) has flag - leaf
    """
    # returns this, as unicode but with flag leaf
    _Node = type("dothier_node", (unicode,), {"leaf":False})

    seq = sorted(seq)
    mem = set()
    for p in seq:
        spl = p.split(sep)
        for ip in xrange(1, len(spl) + 1):
            pp = _Node(sep.join(spl[:ip]))
            if len(spl) == 1 or ip == len(spl):
                pp.leaf = True
            if not pp in mem:
                yield pp
                mem.add(pp)

def dottree(seq, sep='.', fullpath=False):
    """Returns dict of dicts like tree from sequence of dot separated strings.
    For example,
    >>> dottree(["x.y.z", "x.y", "a", "a.b.c", "a.b"])
    {'a': {'b': {'c': {}}}, 'x': {'y': {'z': {}}}}

    and

    >>> dottree(["x.y.z", "x.y", "a", "a.b.c", "a.b"], fullpath=True)
    {'a': {'a.b': {'a.b.c': {}}}, 'x': {'x.y': {'x.y.z': {}}}}
    """
    def ins(map, path):
        d = map
        parents = []
        for p in path:
            if fullpath:
                key = sep.join(parents + [p])
            else:
                key = p
            d = d.setdefault(key, {})
            parents.append(p)
    ret = {}
    seq = sorted(seq)
    for p in seq:
        pp = p.split(sep)
        if not pp:
            continue
        ins(ret, pp)
    return ret

Examples of usage are... in it's docstrings :) They are useful in GUI application: to construct tree or another hierarchical controls from paths (retrieved from config. files, for example).
This code is from my project "HMI pybase" (see Projects).

GFig GIMP 2.6 files parsing/rendering

See modules "gfig" from "HMI pybase" project on Projects page.

Parsing
GFig files are very simple vector format in Gimp 2.x used by GFig plugin for shapes creation. It looks like this:

GFIG Version 0.1
Name: Simly\040face
Version: 0.000000
ObjCount: 1
GridSpacing: 30
GridType: RECT_GRID
DrawGrid: FALSE
Snap2Grid: FALSE
LockOnGrid: FALSE
ShowControl: TRUE
66 150
66 200

Parses only KNOWN SHAPES, all other are ignored. Also parses styles and extra parameters. Suppose fixed structure of file: lines are not joined! Known shapes are:

• line
• rectangle
• circle
• ellipse
• poly
• star
• spiral
• bezier
• arc

Rendering
GFig GIMP 2.6 files renders with Tkinter (on canvas). Supports:

• lines
• circles
• ellipses
• rectangles
• polygons
• arcs
• old style spirals (via lines)

and it's outline-, fill- colors, line width; also polygon has style ruby (recreate canvas object when changed - interesting effect with smooth=1, ruby=True). Not supported: Bezier, spiral, stars. All unknown objects are ignored.

Transforming of objects is supported: move, scale, rotate. It's possible to transform only one object or any selected objects (set) of gfig-file. First, shape class methods are used, for second, GFigRender methods are used. GFigRender transformation methods has arg shapes - shapes to be transformed. Also, you can select GFigRender shapes by find_shapes() method and process it (transform, configure, for ex.). Finding use classname of shape ("CanvasArc", "CanvasEllipse", etc) or any option (Tk cget, gfig option) in **kw form, or name of shape. Usually shapes don't have names (gfig-extension feature), but user can name each shape in order from left-to-right and top-to-bottom (canvas location) with method name_shapes(). find_shapes() returns list, ifind_shapes() - iterator.

Each shape can have label - text short string. Technically, label is created on first labelconfigure(), each other only changes label options. Label is placed by options: side (N|NW|CENTER...) and standard Tk anchor option; so side specified corner (about shape bbox) or center of bbox

Contribs
Thanks for creators of Gimp GFig:

• Andy Thomas
• Spencer Kimball
• Peter Mattis