AWK Literate Programming Personal Tool

LAWK is the simple literate programming tool for personal usage (no collaboration features). For collaboration nanolp can be used see.

Input files are markdown, output - any one. Supports non-ASCII, Windows. Does not support spaces and other special chars in paths. Source is here.

Good feautures are:

• very clean syntax of definition, pasting, writing named chunks
• simple dependencies (awk, mkdir, rm, make) - available for Windows too
• no installation (only copy source somewhere)
• markdown input format
• no special syntax for code chunks
• ctags tag file generation (unsorted)

Roadmap to NanoLP 1.0

1.0 is planned to be beta version. So plans are:

• several new features(for collabortive usage, flex text transformation, etc.)
• tests improvement
• fixing tests on Linux (problem with paths)
• fixing lookup of installed files on Linux (ploblem with paths)
• README files for every test
• presentation and Quick References
• other BUGFIX

And possible another documentation materials...

NanoLP 1.0i released

Special Web edition: now LP code base can be published on the Web as online documentation and at the same time - as library (for including/importing into your LP programs).

Added new functions, new parser, total code refactoring, added events handling mechanism (with chaining support), URL support in filenames, Web-publishing and more...

New Literate Programming Tool: Nano-LP

Yes, yet another and, I thing, last of my attempts. Done WiKi (on project page). Main goal is to keep input file clean and readable (not so cryptic like many other tools) with very "regualar" and simple, intuitive syntax. Main features:

• OpenOffice/LibreOffice
• MarkDown/MultiMarkdown
• Creole
• reStructuredText
• TeX/LaTeX
• txt2tags
• Asciidoc
• HTML/XML
• ... all based on these

Utility is little Python script and supports:

• defining command (macros) with placeholders in the body (code chunk)
• pasting command code chunk
• defining multiple parts code-chunks
• joining, 'ending', etc. several code chunks
• globbing command when paste
• including one file to another (library)
• auto-detecting of cycles
• configurable via simple .INI like file
• works with Python 2.7.x - Python 3.x
• works with Unicode (UTF8)

TODO:

1. Improvements of error reporting: file, line locating, (a special in OO - missed)
2. More tests, better testing
3. Improve Asciidoc parser
4. Realize 1.0 beta
5. ...

New Literate Programming Tool - TCLP 1.0!

I done new LP tool: very small (about 700-800Kb!) and flexible!
Main features are:

1. Document has 'document header' with fields: TITLE, PROJECT, VERSION, DESCRIPTION, AUTHOR, ADDRESS, TAGS, it shown on the top of document.
2. User can define values of this fields (and any other field) and can use defined fields in the document body (even in code chunks). There are special fields: APPNAME, APPVERSION, TIMESTAMP, DATE, TIME.
3. Document has auto-generated TOC (table-of-contents), IT (index-table), ERT (external-references-table). TOC is placed in the start of the document. IT is placed at the end of document. ERT is placed last.
4. TOC is generated automatically for all headers, defined with 'head' command.
5. IT is generated automatically for all code chunks and includes file names of output chunks and lines numbers where to find them.
6. ERT is generated automatically for all external references in the document, defined with 'ref' command and includes back-references to it's usage in the document body.
7. Markup commands includes commands for bold, underlined, italic and monotyped text. Top/bottom indexes are supported also. Markups can be nested.
8. There is special file with abbreviations to be used in section names and acronyms also.
9. User can use acronym in the text from abbreviations file or explicitly with embedded expansion.
10. User can create "vocabularies" - tables of definitions - from abbreviations file or embedded, defined explicitly.
11. There are named counters (auto-incremented) to use in captions and so on.
12. Code chunks are special formatted fragments of source code with lines numbering, optional syntax highlighting (only keywords). Lines can be numbered in continuously manner (with the same counter names) or in independent. User can reference to code lines by chunk name (section) and line number or with defining NAME of code line and usage of this name for referencing.
13. Tables are supported: with attributes (table/row/cell), spanning, nesting.
14. Lists (ordered/unordered) are also supported: with attributes; they can be nested also.
15. User can insert image with 'embed' command. Any fragment of document can be "framed" with 'frame' command. It creates frame with caption (on top/on bottom), name and links (for referencing).
16. There are 2 kinds of citations: inlined and blocked. Citation can have linked URL (shown as "More...")
17. User can insert in document text any previous defined field values, but also variables from environment. Special environment variables is the USER - this is the name of user running tool.
18. There is a special command 'wrap' for wrapping text with any word or quotation symbols, braces, etc.
19. There is a command 'ent' to translate special symbols to HTML entities.
20. There are references:
    • to external URL
    • to header
    • to line of code chunk/named line of code chunk
    • to code chunk
    • to frames
    • to any place of document where user defines own link
21. There are 2 special commands: 'atstart', 'atend'. Mainly used is 'atend' for code generation from code chunks. User can define there what code chunks to be written to output source files (by sections, by glob-pattern of sections). Also is possible to write any free text to output file.
22. Most of created HTML tags has CSS-classes. For all available CSS classes, see 'tclp.css' file. Used CSS file can be defined with command line option.
23. User can add syntax keywords for any other language (for highlighting).

You can try it from Google Projects: http://code.google.com/p/tclp/

New ypweb version

Added new version of ypweb (Literate Programming tool): 1.1a. This is the bugfix of 1.0a version. See http://code.google.com/p/ypweb/. Plans are to:

• add options to command (last pie "..; -h")
• add command line option -h, -h N -- show only N lines of source in doc
• add command ";; ..." -- means hide other lines of source in doc until next "..."
• add option to prev. command ";; ...; -l 10" - hide after 10 lines
• add jumping to tag definition (use ctags) (apart processing doc/src?)
• add unittests

ypweb literate programming tool

After I tried LP technique, I decided that it help me to understand my code better and cuts many possible errors. But I need easier tool, lesser verbose... I implemented very lightweight Python 3 tool (about 20Kb) and called it ypweb. It's hosted on Google.
Main differences with other LP tools are:

• produces only reStructuredText output
• very lightweight (only one Python 3 file) about 20Kb
• doesn't need comment line symbols for document strings (like #!) and code-chunks: only plain text
• document and code context are switched by special line with short variants
• generated index contains used by user (not predefined) classifiers
• doesn't allow code-chunks "placeholders" (defining in one place/inserting in another)
• supports any input and output charsets (see -h)

Short example:

  1 | Some definitions (macroses) used in COM module:
  2 | ;; x.h; defs; COM definitions
  3 | ;; Port number
  4 | #define PORT 1
  5 | ;; Baudrate
  6 | #define BAUD 9600
  7 | ;; funs; open()
  8 | int open(char *name, char *mode);
  9 | ;;
 10 | Another doc strings...
 11 | ;; x.c; funs; open()
 12 | code of open function here...
 13 | ;; x.h; defs; XXX
 14 | #define XXX
 15 | ;;
 16 | docs strings again...

Line 1 is the document chunk (file may starts with any chunk: doc or code). Line 2 is the special kind of line which switch context to code and begins with special marker (;;) and consists of 3 parts (may be lesser) splited by ;. Both symbols (;; and ;) may be changed to another in command line. It "says": next chunk is code, generates x.h file with some definitions ("defs" but may be any word preferred by user) and is the "COM definitions". These fields will be used also in index generating.

Line 3 is short form of switch-context-line (only caption, no file name or classifier). Line 4 is code and will be saved in the source file. Line 4 is like line 3, next is source line again...

Line 7 is short switch-context-line too and consists of classifier and caption, no filename, so will be saved in the same file - x.h.

Line 9 is empty switch-context-line, so it ends code chunk, nothing else. Next is always doc chunk.
Line 11 is switch-context-line which starts new source file: x.c. Line 13 begins code chunk for early created x.h file. Well, switch-context-line switch into code chunk and defines what is the code if not empty, empty - switches into doc chunk.

It's very clean and simple format. And is the reStructuredText too. So, you can use in it any kind of usual reSt markup!

After processing this file, .rst file and x.h, x.c will be generated. Also .rst file will have index section in the end with references to definitions of the code chunks.

Another approach to Literate Programming

I tried for Literate Programming (LP) two solutions: Nuweb and Python tool Pyweb and found Pyweb tool is better. But with LP there are some problems:

1. it's not very easy to work with "hidden" source: after tangling http://www.blogger.com/img/blank.gifand weaving you get pure source code and should debug source code (this is usually a "solid" process), after this - should modify .w file


2. usage of "placeholders" lines split code (difficult for debugging)

Another approach is to use reStructuredText document generator system (see docutils). How to? It's easy: instead of embedding into source code the hyper-text formatting, images, links, you can write your program like story (in usual LP style), use RST syntax (like WiKi) for images embedding and various styles - your code is Literate Programming story as before. But now you can embed code fragments where you need with "include" directive of docutils. This directive requires "startd-after" and "end-before" options for this case. These options defines line (string fragment) the code starts after, and line - the code ends before. I use for this lines Vim foldering lines. See:

// Source code:
int x;
// Opening port {{{
int open_port(int port) {
...
}
// }}}

This is very usuable for folding big source code!
And in your .rst file (which plays a role of the .w file but for weaving only):

and we define the function for a port opening:

.. include:: port.c
    :start-after: // Opening port {{{
    :end-before: // }}}

Thos technique is not pure LP but is like LP without LP cons. And it's simple documentation approach :)