The curriculum build specification

First, we may want to create the course core in multiple natural languages, e.g., English (en-us) or Spanish (es-mx). To accommodate this, the source dir pathway/core has a langs subdir, with further subdirs for each natlang that we want: i.e., en-us and es-mx. The source tree thus looks like:

We’ll call the pathway/core/langs/en-us the core pathway’s root directory (for the en-us natlang).

By default, the programming language associated with a pathway is Pyret.

We may also want to create the courses for our core pathway for multiple programming languages, e.g., pyret and wescheme. To accommodate this, we include the file proglang.txt in the pathway root directory (pathways/core/langs/en-us/), and list in it the various proglangs supported. Thus the contents of proglang.txt will look like:

Sometimes we have a pathway that has associated with it only a single proglang, but that proglang is not Pyret. As an example, consider the pathway shipwrecks which is defined for the proglang spreadsheets. In this case, we still need a proglang.txt file containing the single line

The pathway root directory has a file index.adoc, which specifies the pathway narrative. The pathway root also has a resources/ subdirectory that houses additional documentation that is specific to teachers. The pathway root may also contain subdirs for front-matter, back-matter, and images: the former two add flanking matter to the final workbook for the pathway’s course, while images is a convenient location for image subfiles.

The pathway root directory has a file lesson-order.txt listing the names of the lessons associated with it, in order. These lessons occur as directories in the lessons/ directory in the repo top dir.

As with pathways, lessons can be in different natlangs and proglangs. To accommodate multiple natlangs, each lesson has a langs subdir, with subdirs (e.g., en-us, es-mx) for the different natlangs. (This is similar to how we specify multiple natlangs for pathways.)

We’ll call lessons/simple-data-types/langs/en-us the lesson root directory.

As for pathways, a lesson root directory can contain a proglang.txt listing the proglang(s) associatable with it. If no proglang.txt is present, the (single) proglang is assumed to be pyret.

If more than one proglang is needed, or if the single proglang is something other than pyret, list it in proglang.txt.

There are also some source files that are not affiliated with any particular pathway or lesson. These are kept in shared/langs/en-us/docroot (and the like, for other natlangs). Chief among these are textbook descriptions which are .adoc files in the textbooks/ subdir.

The courses and lessons are not direct subdirs of distribution, in order to accommodate different versions of pathways and lessons based on the natlang used. To effect this, the distribution has subdirs for each natlang (called en-us, es-mx, etc.), and it is in the courses and lessons subdirs of these that we find the natlang-specific courses and lessons. Thus, the two natlang versions of the pathway core are laid out as follows:

Note that we use the same name core for the two natlang versions in the distribution. We can do this because they reside inside two different subtrees distribution/en-us distribution/es-mx and thus don’t risk a name clash.

As with pathways, so with lessons: To accommodate multiple natlangs, the lessons go into the lessons subdir of their natlang. E.g., for the lesson simple-data-types, we have:

In the following, we will assume the prevailing natural language is en-us, so we will restrict our attention to the distribution/en-us subtree. For other natlangs, the setup is exactly the same but in the appropriate natlang-based subtree.

Also, we will call distribution/en-us/courses/core the course root directory (of the English version of the core pathway). The dir distribution/en-us/lessons/simple-date-types is the distribution lesson root directory (of the English version of the simple-data-types lesson).

Typically — as seen in the above example — the built names of the pathway and the lesson are the same as their names in the source, but not always.

By default, the programming language associated with a course is pyret. This is marked in the course root directory as the 0-length file .cached/.proglang-pyret. (I.e., distribution/en-us/courses/core/.cached/.proglang-pyret.)

But we as have seen, a pathway may have a proglang.txt listing a different or multiple proglangs.

Thus, for the pathway core, two different courses are created in distribution/en-us/courses. We continue to use the unadorned pathway name, core, for the Pyret version. For other proglangs, the course dir’s name uses the proglang as a hyphenated suffix. Thus for the WeScheme version, the course is entitled core-wescheme. I.e., the two course root directories for core are

These two directories have their own proglang marker files, i.e., .cached/.proglang-pyret (as before) and .cached/.proglang-wescheme.

We often have pathways whose proglang.txt contains only a single proglang, but that proglang isn’t Pyret. E.g., the pathway shipwrecks whose proglang.txt contains only spreadsheets. In such a case, the built course is simply called shipwrecks rather than shipwrecks-spreadsheets, since there is no Pyret version to distinguish it from.

As with pathways, lessons can be in different proglangs, also specified with proglang.txt. If no proglang.txt is present, the (single) proglang is assumed to be pyret. The distribution version of the non-Pyret lessons have the proglang as hyphenated suffix, e.g.,

Again, as with pathways, a proglang-marker file .cached/.proglang-wesceme is placed in the distribution lesson root to identify it as a lesson using wescheme.

Note that the pathway mentions in its lesson-order.txt the lesson names in unadorned form (no proglang suffix). On building, a pathway becomes a course, and the distribution lessons associated with it are the right versions corrected for proglang. Thus the pathway core has simple-data-types as a constituent lesson. Moving over to the built distribution, the course core (Pyret) includes the lesson simple-data-types (Pyret), and the course core-wescheme includes the lesson simple-data-types-wescheme.

The pathway narrative index.adoc in the pathway root eventually will get converted to index.shtml in the course root. The various .adoc files in the front-matter, back-matter and resources subdirs also will get converted to HTML files, with suffix .shtml at the top level and .html at lower depths, i.e., in any pages and solution-pages subdirs. However, these HTML conversions aren’t performed at this time.

The lesson plan index.adoc in the lesson root eventually will get converted to index.shtml in the distribution lesson root.

The .adoc files in the pages/ and solution-pages/ subdirs along with any .adoc files they include will get converted to .html files alongside.

The pathway-independent files in shared/langs/en-us/docroot are copied over to distribution/en-us.

The first phase initializes the distribution directory, then copies the lessons and pathways over, adjusting subdirectories, and creating proglang-specific copies as needed. It sets up the temp files that will contain the input date to run various conversions in batch mode. Two specific PDFs, the page-not-found and the bilingual glossary, are requested to start the ball rolling.

The second phase does the bulk of the conversions. Requests are set up in the batch-input files for passing the various adoc files to the Racket preprocessor, which will create corresponding asc files. These are then converted to html files via a batch call to Asciidoctor.

Different batch files are then used to pick up primitives, the lesson inter-dependencies, and global image and pathway-toc listings, and to add some postprocessing.

Finally the postprocessed HTML files are converted to PDFs, and these are assembled into workbook pages.

After copying a pathway into its distribution lesson directory with fixed proglang, the build process drills down its directories to see if there are any proglang-specific shadowing subdirectories. These are named for the proglang, e.g., pyret, wescheme, etc. The contents of the relevant proglang subdir are copied to the containing directory. All the other proglang subdirs are deleted. This ensures that all the files in the course directory, at whatever depth, have content appropriate to its proglang.

Next, if a subdirectory called pages occurs at any level — typically these are in the front-matter, back-matter and resources subdirs --, the build ensure that a subdir called solution-pages is alongside it. Furthermore, it ensures that its contents are the same as the pages subdir, except for such files that are pre-existent in it (from the repo).

Next the build looks for workbook-pages.txt inside these pages subdir. A sanitized version — removing comments — of this is placed in .cached/.workbook-pages.txt.kp, and an even sparer version — removing landscape/portrait information — is placed in .cached/.workbook-pages-ls.txt.kp.

Course dirs undergo the same modifications above for proglang and solution-pages. Note here that the subdirs front-matter, back-matter, and resources may contain pages/ subdirs with the appropriate workbook-pages.txt, and these are mined to get .cached/.workbook-pages.txt.kp and .cached/.workbook-pages-ls.txt.kp as well, just as for the lessons.

The names of the pathway-independent .adoc files under distribution/en-us/textbooks are collected.

The .adoc file, together with its #:containing-directory and #:dist-root-dir value is added to $ADOCABLES_INPUT.

The .asc file (which sits in the .cached subdir), pathname relative to distribution/en-us is added to $ADOC_INPUT.

The .html file (not yet postprocessed, sits in .cached), pathname relative to distribution/en-us) is added to $ADOC_POSTPOC_PWYINDEP_INPUT.

We collect all the workbook-page .adoc files and add them to $ADOCABLES_INPUT. The #:containing-directory, #:dist-root-dir, #:lesson, #:other-dir, #:solutions-mode? and #:proglang values are included.

#:lesson is simply the basename of the distribution lesson.

#:other-dir is a boolean identifying if the file is simply meant for inclusion in other adoc files and not for full-scale conversion themselves. Such files are located in subdirs named fragments, xtra, or xtras.

#:solution-mode? is a boolean set to true only for adoc files in solution-pages. Such files contain information meant for teachers but not students.

The .asc version of the file is added to $ADOC_INPUT.

The cached .html version of the file is added ot $ADOC_POSTPROC_WORKBOOKPAGE_INPUT.

We collect all the lesson-plan .adoc files and add them to $ADOCABLES_INPUT. The #:containing-directory, #:dist-root-dir, #:lesson-plan, #:proglang, and #:other-proglangs values are included.

#:lesson-plan is simply the basename of the distribution lesson.

#:other-proglangs is a list of the other proglangs for which this lesson is available. This is used to have the lesson plan include links to the plans for the same lesson in the other proglangs.

The .asc version of the file is added to $ADOC_INPUT.

The cached .html version of the file is added ot $ADOC_POSTPROC_LESSONPLAN_INPUT.

Collect all the .adoc files except the narrative file and add them to $ADOCABLES_INPUT. Other values included: #:containing-directory, #dist-root-dir, #:other-dir, #:resources, #:target-pathway, #:solutions-mode?, #:proglang.

#:resources is set to true.

#:target-pathway is the (base) name of the course directory (with the proglang suffix, if any).

The .asc file is added to $ADOC_INPUT.

The cached .html file is added to $ADOC_POSTPROC_RESOURCES_INPUT.

Collect all the narrative .adoc files in the distribution and add then to $ADOCABLES_INPUT. Other values included: #:containing-directory, #:dist-root-dir, #:narrative, #:target-pathway, #:proglang, #:other-proglangs.

#:narrative is set to true.

#:other-proglangs is the list of other proglangs for which this course is available, so they can link to each other.

The .asc file is added to $ADOC_INPUT.

The cached .html file is added to $ADOC_POSTPROC_NARRATIVE_INPUT.

We also the .asc file for the pathway glossary (this is generated by the Racket preprocessor) to $ADOC_INPUT. Its cached .html file is added to $ADOC_POSTPROC_NARRATIVEAUX_INPUT.

After all the $ADOC…​ batch files have been populated, the build runs a Racket preprocessor on the entries in $ADOCABLES_INPUT to create the .cached/*.asc files.

This is because the .adoc files created by the curriculum authors contains some directives in addition to the commands provided by raw AsciiDoc. A Racket program preprocesses these away to create a raw AsciiDoc file. This has the extension .asc and is situated in a .cached subdir alongside the .adoc file penned by the author.

The additional keyword values provided in the $ADOCABLES_INPUT enables the Racket preprocessor to correctly address the different types of .adoc files we have (workbook pages, lesson plans, narratives, etc.).

A single Racket call is sufficient to process all the entries in $ADOCABLES_INPUT.

Once the Racket preprocessor done, we have a bunch of .cached/*.asc files in the distribution, and the list of these we have already captured in the file $ADOC_INPUT.

These files are now ready to be processed by Asciidoctor to produce the corresponding .html.

The asciidoctor command can take a bunch of input files as command-line input, so we pass it the contents of $ADOC_INPUT.

The lesson dependency graph dependency-graph.js is generated by the program make-dependency-graph.lua. It shows, for each lesson:

The primitives are generated for each lesson as .cached/.index-primitives.txt.kp by the program collect-primitives.lua.

Note that the JS file represent the collected information as a JSON object set to a variable. The file itself isn’t JSON. This also holds for the next two JS files described below.

A glossary of image information, images.js, is compiled from the images/lesson-imags.json file in each lesson.

A list of all the courses with their constituent lessons is generated in pathway-tocs.js.

In addition to these JS files, a bilingual glossary bilingual-glossary.html is created in distribution/en-us/lib. This is a browsable version of the file glossary-terms.rkt in the repo.

The script make-pdf.sh passes the list of HTML files generated in $PUPPETEER_INPUT to the Node program html2pdf.js, creating the corresponding PDFs.

The program make-workbook-jsons.lua scans the course and lesson dirs for various temp files create the previous portions of the build to create JSON files that list the constituent PDFs for each type of workbook.

The script make-books.sh uses these JSON files to create the final workbook PDFs.

Makefile.phase1 initializes the distribution/ directory, with subdir en-us/, which has subdirs lib/, extlib/, lessons/, courses/. It also zeroes out the various temp files in en-us/.cached, which contain lists of files that will eventually fed as a batch to various processing scripts. We will mention these batch files as they come up in the process.

Using generated make rules, distribution/en-us/courses/ is populated with copies of the pathways, and distribution/en-us/lessons/ with copies of the lessons. If a lesson allows multiple proglangs (e.g., pyret, wescheme, codap, etc.), it is duplicated for each such proglang, with the proglang added as a suffix, except for pyret and none, which do not get a suffix.

The courses and lessons in the distribution are "massaged" to create solution-pages/ alongside any pages/ subdirs, and to move any shadowing files specifically meant for the prevailing proglang to overwrite files that are generic or meant for other proglangs. The massaging is done by two external scripts: massage-distribution-lesson.sh and massage-course.sh. Both these scripts make use of a program collect-workbook-pages.lua to identify within each lesson (or lesson-like entity like front-matter and back-matter in a course) all the pages that are eligible to go into the workbook — the so-called workbook pages.

After populating (or updating) the en-us/{courses,lessons}, Makefile.phase1 collects all the exercises in the lessons (these are specific exercise directives used in the adoc source of the lessons). These are placed in .cached/ subdirs in the individual lesson pages/.

In addition, phase 1 also takes care of creating the HTML version of the bilingual glossary file, and adds it to the batch file $PUPPETEER_INPUT. $PUPPETEER_INPUT will eventually contain all the HTML files that will need to be converted into PDF pages.

After phase 1 is done, Makefile.phase2 picks up the next phase. It identifies the different kinds of .adoc files (already copied under {lessons, courses} in the distribution, and creates make rules for converting these to .asc files using the Racket-based preprocessor. This is accomplished by updating three batch files: $ADOCABLES_INPUT, $ADOC_INPUT and a third $ADOC_POSTPROC_*_INPUT file whose name depends on whether the adoc file in question is a lesson plan, a pathway narrative file, a pathway glossary file, a pathway resources file, a pathway-independent file, or a workbook page (which can be in both lessons and pathways).

After these batch files are updated, an external script run-asciidoctor.sh converts the files in $ADOCABLES_INPUT to their corresponding .asc (also an asciidoc file), using adocables-preproc.rkt, an adoc preprocessor written in Racket. It then uses the $ADOC_INPUT batch file containing the list of .asc’s as input to Asciidoctor. A set of .html files results alongside the .asc’s.

The preprocessing also collects the primitive functions used (if any) in each of the lesson pages, and notes down the lesson prerequisites for each lesson.

If the make var BOOK is set, the preproc rules include a further set of rules that add lines to $PUPPETEER_INPUT, so that the HTML files posited by the preproc (but only finally created after the postproc) will also become candidates for PDF conversion.

This completes the preproc subphase. After it is done, the following bunch of other rules come into play (in no temporal order):

After the primitive generation rule (1 above) is done, a rule calls an external script make-dependency-graph.lua that gathers the generated info about the lesson primitives and prereqs to create a global lesson dependency graph in en-us/dependency-graph.js.

An external script make-images-js.lua collects all the image information from the lessons into a global image glossary at en-us/images.js.

After the postproc rule (2 above) is done, if the make variable BOOK is set, the batch file $PUPPETEER_INPUT is used by a Node program html2pdf.js to create all the individual page PDFs.

After these individual PDFs are available, and again only if BOOK is set, a make rule uses the external program make-workbook-pages.lua to go into each course, collects all its constituent lessons' workbook and exercise pages into six course-specific list of pages for the six different types of workbooks. The same rule then calls the Node program makeWorkbook.js to create each course’s set of workbook PDFs.

By default, make won’t create the PDF versions of the HTML files, nor the collections of them that form the various workbooks. PDF and workbook construction can be time-consuming and so is best relegated to the final run, when the author is sure that the HTML conversions have been thoroughly debugged. Once they are, and the PDFs are desired, request the target book on the make command-line, i.e.,

By default, the checking of the various internal and external links in the documents are not checked — because it can be a time-consuming process. (External-link checking is particularly heinous because it takes a long while for each URL probe to return.)

To enable link checking, use the target linkcheck:

It is often advisable to do the link check just for its own sake, and only once in a while, after the distribution dir is already in place from a previous make.

To remove a previously made distribution entirely, do

The next make — with whatever options you desire — will be on a fresh slate.

To deploy a built distribution to the website, do

Deployment is described in more detail in the repo’s README.