From ee55161508c35dc61633da69b208c248865a5a09 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:13:02 -0700 Subject: [PATCH 01/26] doc: Use `examples` forms in illustrating variable scope --- qi-doc/scribblings/forms.scrbl | 52 ++++++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/qi-doc/scribblings/forms.scrbl b/qi-doc/scribblings/forms.scrbl index 0656c5927..25edc9379 100644 --- a/qi-doc/scribblings/forms.scrbl +++ b/qi-doc/scribblings/forms.scrbl @@ -836,32 +836,60 @@ A form of generalized @racket[sieve], passing all the inputs that satisfy each @subsection{Variable Scope} -In general, bindings are scoped to the @emph{outermost} threading form (as the first example above shows), and may be referenced downstream. We will use @racket[(gen v)] as an example of a flow referencing a binding, to illustrate variable scope. +We will use @racket[(gen v)] as an example of a flow referencing a binding, to illustrate variable scope. -@codeblock{(~> 5 (as v) (gen v))} +In general, bindings are scoped to the @emph{outermost} threading form, and may be referenced downstream. -... produces @racket[5]. +@examples[ + #:eval eval-for-docs + #:label #f + (~> (5) (as v) (gen v)) + (~> (5) (-< (~> sqr (as v)) + _) (gen v)) +] A @racket[tee] junction binds downstream flows in a containing threading form, with later tines shadowing earlier tines. -@codeblock{(~> (-< (~> 5 (as v)) (~> 6 (as v))) (gen v))} - -... produces @racket[6]. +@examples[ + #:eval eval-for-docs + #:label #f + (~> () (-< (~> 5 (as v)) + (~> 6 (as v))) (gen v)) +] A @racket[relay] binds downstream flows in a containing threading form, with later tines shadowing earlier tines. -@codeblock{(~> (gen 5 6) (== (as v) (as v)) (gen v))} - -... produces @racket[6]. +@examples[ + #:eval eval-for-docs + #:label #f + (~> (5 6) + (== (as v) + (as v)) + (gen v)) +] In an @racket[if] conditional form, variables bound in the condition bind the consequent and alternative flows, and do not bind downstream flows. -@codeblock{(if (~> ... (as v) ...) (gen v) (gen v))} +@examples[ + #:eval eval-for-docs + #:label #f + (on ("Ferdinand") + (if (-< (~> string-titlecase (as name)) + (string-suffix? "cat")) + (gen name) + (gen (~a name " the Cat")))) +] Analogously, in a @racket[switch], variables bound in each condition bind the corresponding consequent flow. -@codeblock{(switch [(~> ... (as v) ...) (gen v)] - [(~> ... (as v) ...) (gen v)])} +@examples[ + #:eval eval-for-docs + #:label #f + (switch ("Ferdinand the cat") + [(-< (~> string-titlecase (as name)) + (string-suffix? "cat")) (gen name)] + [else "dog"]) +] As @racket[switch] compiles to @racket[if], technically, earlier conditions bind all later switch clauses (and are shadowed by them), but this is considered an incidental implementation detail. Like @racket[if], @racket[switch] bindings are unavailable downstream. From eeaf349fdd6b47ccd49b01f513c859b5c40dd173 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:14:41 -0700 Subject: [PATCH 02/26] doc: illustrate being intentional about effects with an example --- qi-doc/scribblings/field-guide.scrbl | 32 +++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 5f9407040..4e756121d 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -31,7 +31,37 @@ A journeyman of one's craft -- a woodworker, electrician, or a plumber, say -- a @subsection{Be Intentional About Effects} -Qi encourages a style that avoids "accidental" effects. A flow should either be pure (that is, it should be free of "side effects" such as printing to the screen or writing to a file), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. +Qi encourages a style that avoids "accidental" effects. + +In functional programming, "effects" refer to anything that the function does that is not captured in its inputs and outputs. This could include things like printing to the screen or writing to a file. + +A flow should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. + +For example, say that we wish to perform a simple numeric transformation on an input number, but also print the intermediate values to the screen. We might do it this way: + +@examples[ + #:eval eval-for-docs + #:label #f + (define (my-square v) + (displayln v) + (sqr v)) + + (define (my-add1 v) + (displayln v) + (add1 v)) + + (~> (3) my-square my-add1) +] + +This is considered poor style since we've mixed pure functions with implicit effects. Instead, following the above guideline, we would write it this way: + +@examples[ + #:eval eval-for-docs + #:label #f + (~> (3) (ε displayln sqr) (ε displayln add1)) +] + +This uses the pure functions @racket[sqr] and @racket[add1], extracting the effectful @racket[displayln] as an explicit @racket[effect]. @section{Debugging} From dd5280d801a6b4a84d8c2116219f34fad32e9448 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:20:26 -0700 Subject: [PATCH 03/26] doc: clarify order of effects wrt `esc` and `effect` --- qi-doc/scribblings/field-guide.scrbl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 4e756121d..eefc37152 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -396,6 +396,8 @@ Yet, either implementation produces the same output: @racket[(list 1 9 25)]. So, to reiterate, while the output of Qi flows will be the same as the output of equivalent Racket expressions, they may nevertheless exhibit a different order of effects. +If you'd like to ensure a specific order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, define your flow using @racket[esc] (although this would lose any Qi compiler optimizations). + @section{Effectively Using Feedback Loops} @racket[feedback] is Qi's most powerful looping form, useful for arbitrary recursion. As it encourages quite a different way of thinking than Racket's usual looping forms do, here are some tips on "grokking" it. From 34cac7ff1e06b21bbfa2a295984e1d859dadd4fc Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:20:55 -0700 Subject: [PATCH 04/26] doc: don't quote "deforested" --- qi-doc/scribblings/field-guide.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index eefc37152..3763e71ed 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -390,7 +390,7 @@ So in general, use mutable values with caution. Such values can be useful as sid Consider the Racket expression: @racket[(map sqr (filter odd? (list 1 2 3 4 5)))]. As this invokes @racket[odd?] on all of the elements of the input list, followed by @racket[sqr] on all of the elements of the intermediate list, if we imagine that @racket[odd?] and @racket[sqr] print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence @racket[1,2,3,4,5,1,3,5]. -The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr))]. As this sequence is @seclink["Don_t_Stop_Me_Now"]{"deforested" by Qi's compiler} to avoid multiple passes over the data and the memory overhead of intermediate representations, it invokes the functions in sequence @emph{on each element} rather than @emph{on all of the elements of each list in turn}. The printed sequence with Qi would be @racket[1,1,2,3,3,4,5,5]. +The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr))]. As this sequence is @seclink["Don_t_Stop_Me_Now"]{deforested by Qi's compiler} to avoid multiple passes over the data and the memory overhead of intermediate representations, it invokes the functions in sequence @emph{on each element} rather than @emph{on all of the elements of each list in turn}. The printed sequence with Qi would be @racket[1,1,2,3,3,4,5,5]. Yet, either implementation produces the same output: @racket[(list 1 9 25)]. From b324988c432c7436055adc3e5465355f99480302 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:42:51 -0700 Subject: [PATCH 05/26] doc: minor wording --- qi-doc/scribblings/field-guide.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 3763e71ed..88cdd06ad 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -396,7 +396,7 @@ Yet, either implementation produces the same output: @racket[(list 1 9 25)]. So, to reiterate, while the output of Qi flows will be the same as the output of equivalent Racket expressions, they may nevertheless exhibit a different order of effects. -If you'd like to ensure a specific order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, define your flow using @racket[esc] (although this would lose any Qi compiler optimizations). +If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, define your flow using @racket[esc] (although this would lose any Qi compiler optimizations). @section{Effectively Using Feedback Loops} From 2a8960f1b3e7525a2a55d279b2668807ba68b5db Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:43:17 -0700 Subject: [PATCH 06/26] document another common error for "bad syntax" --- qi-doc/scribblings/field-guide.scrbl | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 88cdd06ad..cca6bc22b 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -202,9 +202,16 @@ Methodical use of @racket[gen] together with the @seclink["Using_a_Probe"]{probe ; in: lambda } -@bold{Meaning}: The Racket interpreter received syntax, in this case simply "lambda", that it considers to be invalid. Note that if it received something it didn't know anything about, it would say "undefined" rather than "bad syntax." Bad syntax indicates known syntax used in an incorrect way. +@bold{Meaning}: The expander (either the Racket or Qi expander) received syntax, in this case simply "lambda", that it considers to be invalid. Note that if it received something it didn't know anything about, it would say "undefined" rather than "bad syntax." Bad syntax indicates known syntax used in an incorrect way. -@bold{Common example}: A Racket expression has not been properly escaped within a Qi context. For instance, @racket[(flow (lambda (x) x))] is invalid because the wrapped expression is Racket rather than Qi. To fix this, use @racket[esc], as in @racket[(flow (esc (lambda (x) x)))]. +@bold{Common example}: A Racket expression has not been properly escaped within a Qi context. For instance, @racket[(☯ (lambda (x) x))] is invalid because the wrapped expression is Racket rather than Qi. To fix this, use @racket[esc], as in @racket[(☯ (esc (lambda (x) x)))]. + +@codeblock{ +; not: bad syntax +; in: not +} + +@bold{Common example}: Similar to the previous one, a Racket expression has not been properly escaped within a Qi context, but in a special case where the Racket expression has the same name as a Qi form. In this instance, you may have used @racket[(☯ not)] expecting to invoke Racket's @racket[not] function, since @seclink["Identifiers"]{function identifiers may be used as flows directly} without needing to be escaped. But as Qi has a @racket[not] form as well, Qi's expander first attempts to match this against legitimate use of Qi's @racket[not], which fails, since this expects a flow as an argument and cannot be used in identifier form. To fix this, use an explicit @racket[esc], as in @racket[(☯ (esc not))]. @bold{Common example}: Trying to use a Racket macro (rather than a function), or a macro from another DSL, as a @tech{flow} without first registering it via @racket[define-qi-foreign-syntaxes]. In general, Qi expects flows to be functions unless otherwise explicitly signaled. From bc0835fb2e7c196da49fbe779a6bc53243a4e3a2 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:44:13 -0700 Subject: [PATCH 07/26] doc: update bindings and nonlinearity section for `as` bindings --- qi-doc/scribblings/field-guide.scrbl | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index cca6bc22b..5ebcfe396 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -502,9 +502,9 @@ Using this approach, you would need to register each such foreign macro using @r @subsection{Bindings are an Alternative to Nonlinearity} -In some cases, we'd prefer to think of a nonlinear @tech{flow} as a linear sequence on a subset of arguments that happens to need the remainder of the arguments somewhere down the line. In such cases, it is advisable to employ bindings so that the flow can be defined on this subset of them and employ the remainder by name. +In some cases, we'd prefer to think of a nonlinear @tech{flow} as a linear sequence on a subset of arguments that happens to need the remainder of the arguments somewhere down the line. In such cases, it is advisable to employ @seclink["Binding"]{bindings} so that the flow can be defined on this subset of them and employ the remainder by name. -For example, these are equivalent: +For example, for a function called @racket[make-document] accepting two arguments that are the name of the document and a file object, these implementations are equivalent: @codeblock{ (define-flow make-document @@ -516,8 +516,8 @@ For example, these are equivalent: } @codeblock{ - (define (make-document name file) - (~>> (file) + (define-flow make-document + (~>> (== (as name) _) file-contents (parse-result document/p) △ From 01023df0a7805863a4f128ac5c0c674cc029d5af Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 15 Jan 2024 16:52:33 -0700 Subject: [PATCH 08/26] doc: another tip - use `NOT` instead of Racket's `not` --- qi-doc/scribblings/field-guide.scrbl | 2 +- qi-doc/scribblings/forms.scrbl | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 5ebcfe396..246a3e520 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -211,7 +211,7 @@ Methodical use of @racket[gen] together with the @seclink["Using_a_Probe"]{probe ; in: not } -@bold{Common example}: Similar to the previous one, a Racket expression has not been properly escaped within a Qi context, but in a special case where the Racket expression has the same name as a Qi form. In this instance, you may have used @racket[(☯ not)] expecting to invoke Racket's @racket[not] function, since @seclink["Identifiers"]{function identifiers may be used as flows directly} without needing to be escaped. But as Qi has a @racket[not] form as well, Qi's expander first attempts to match this against legitimate use of Qi's @racket[not], which fails, since this expects a flow as an argument and cannot be used in identifier form. To fix this, use an explicit @racket[esc], as in @racket[(☯ (esc not))]. +@bold{Common example}: Similar to the previous one, a Racket expression has not been properly escaped within a Qi context, but in a special case where the Racket expression has the same name as a Qi form. In this instance, you may have used @racket[(☯ not)] expecting to invoke Racket's @racket[not] function, since @seclink["Identifiers"]{function identifiers may be used as flows directly} without needing to be escaped. But as Qi has a @racket[not] form as well, Qi's expander first attempts to match this against legitimate use of Qi's @racket[not], which fails, since this expects a flow as an argument and cannot be used in identifier form. To fix this in general, use an explicit @racket[esc], as in @racket[(☯ (esc not))]. In this specific case, you could also use Qi's @racket[(☯ NOT)] instead. @bold{Common example}: Trying to use a Racket macro (rather than a function), or a macro from another DSL, as a @tech{flow} without first registering it via @racket[define-qi-foreign-syntaxes]. In general, Qi expects flows to be functions unless otherwise explicitly signaled. diff --git a/qi-doc/scribblings/forms.scrbl b/qi-doc/scribblings/forms.scrbl index 25edc9379..bd5bc03e8 100644 --- a/qi-doc/scribblings/forms.scrbl +++ b/qi-doc/scribblings/forms.scrbl @@ -216,7 +216,7 @@ The core syntax of the Qi language. These forms may be used in any @tech{flow}. @defidform[NOT] @defidform[!] )]{ - A Boolean NOT gate, this negates the input. + A Boolean NOT gate, this negates the input. This is equivalent to Racket's @racket[not]. @examples[ #:eval eval-for-docs From 6b258bb68da17c913f8152931f9d1eb649942547 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 16 Jan 2024 16:58:32 -0700 Subject: [PATCH 09/26] doc: link to section describing deforestation --- qi-doc/scribblings/intro.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/intro.scrbl b/qi-doc/scribblings/intro.scrbl index 280c2f0c9..86adb119c 100644 --- a/qi-doc/scribblings/intro.scrbl +++ b/qi-doc/scribblings/intro.scrbl @@ -73,7 +73,7 @@ For macros, we cannot use them naively as @tech{flows} because macros expect all The threading library also provides numerous shorthands for common cases, many of which don't have equivalents in Qi -- if you'd like to have these, please @hyperlink["https://github.com/drym-org/qi/issues/"]{create an issue} on the source repo to register your interest. -Finally, by virtue of having an optimizing compiler, Qi also offers performance benefits in some cases, including for use of sequences of standard functional operations on lists like @racket[map] and @racket[filter], which in Qi avoid constructing intermediate representations along the way to generating the final result. +Finally, by virtue of having an optimizing compiler, Qi also offers performance benefits in some cases, including for use of sequences of standard functional operations on lists like @racket[map] and @racket[filter], which in Qi @seclink["Don_t_Stop_Me_Now"]{avoid constructing intermediate representations} along the way to generating the final result. @close-eval[eval-for-docs] @(set! eval-for-docs #f) From 35761a9a5b1b89dede9b684ecc8576e45d7e2d0b Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 16 Jan 2024 19:21:46 -0700 Subject: [PATCH 10/26] doc: clarify that `as` produces no output --- qi-doc/scribblings/forms.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/forms.scrbl b/qi-doc/scribblings/forms.scrbl index bd5bc03e8..33083e496 100644 --- a/qi-doc/scribblings/forms.scrbl +++ b/qi-doc/scribblings/forms.scrbl @@ -819,7 +819,7 @@ A form of generalized @racket[sieve], passing all the inputs that satisfy each @section{Binding} @defform[(as v ...)]{ - A @tech{flow} that binds an identifier @racket[v] to the input value. If there are many input values, than there should be as many identifiers as there are inputs. + A @tech{flow} that binds an identifier @racket[v] to the input value. If there are many input values, than there should be as many identifiers as there are inputs. Aside from introducing bindings, this flow produces no output. @examples[ #:eval eval-for-docs From 5eb80ff2db6865ed8dfb2cecec8f830d41930541 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 16 Jan 2024 19:23:03 -0700 Subject: [PATCH 11/26] doc: minor improvements and crosslinks --- qi-doc/scribblings/field-guide.scrbl | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 246a3e520..714b8253e 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -33,9 +33,9 @@ A journeyman of one's craft -- a woodworker, electrician, or a plumber, say -- a Qi encourages a style that avoids "accidental" effects. -In functional programming, "effects" refer to anything that the function does that is not captured in its inputs and outputs. This could include things like printing to the screen or writing to a file. +In functional programming, "effects" refer to anything that the function does that is not captured in its inputs and outputs. This could include things like printing to the screen, writing to a file, or mutating a global variable. -A flow should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. +A @tech{flow} should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. For example, say that we wish to perform a simple numeric transformation on an input number, but also print the intermediate values to the screen. We might do it this way: @@ -393,7 +393,7 @@ So in general, use mutable values with caution. Such values can be useful as sid @subsubsection{Order of Effects} - Qi flows may exhibit a different order of effects (in the functional programming sense) than equivalent Racket functions. + Qi @tech{flows} may exhibit a different order of effects (in the @seclink["Be_Intentional_About_Effects"]{functional programming sense}) than equivalent Racket functions. Consider the Racket expression: @racket[(map sqr (filter odd? (list 1 2 3 4 5)))]. As this invokes @racket[odd?] on all of the elements of the input list, followed by @racket[sqr] on all of the elements of the intermediate list, if we imagine that @racket[odd?] and @racket[sqr] print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence @racket[1,2,3,4,5,1,3,5]. From a4e25b5ca1b7fc747512c9d373def11c6aafcc5d Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Sat, 20 Jan 2024 18:32:47 -0700 Subject: [PATCH 12/26] doc: Qi's stratified architecture and Syntax Spec --- qi-doc/scribblings/macros.scrbl | 2 +- qi-doc/scribblings/principles.scrbl | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/macros.scrbl b/qi-doc/scribblings/macros.scrbl index 96550aa43..39e785d01 100644 --- a/qi-doc/scribblings/macros.scrbl +++ b/qi-doc/scribblings/macros.scrbl @@ -14,7 +14,7 @@ Qi may be extended in much the same way as Racket -- using @tech/reference{macros}. Qi macros are indistinguishable from built-in Qi forms during the macro expansion phase, just as user-defined Racket macros are indistinguishable from macros that are part of the Racket language. This allows us to have the same syntactic freedom with Qi as we are used to with Racket, from being able to @seclink["Adding_New_Language_Features"]{add new language features} to implementing @seclink["Writing_Languages_in_Qi"]{entire new languages} in Qi. -This "first class" macro extensibility of Qi follows the general approach described in @hyperlink["https://dl.acm.org/doi/abs/10.1145/3428297"]{Macros for Domain-Specific Languages (Ballantyne et. al.)}. +For more on how this is accomplished under the hood, see @secref["It_s_Languages_All_the_Way_Down"]. @table-of-contents[] diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 0922eb7fa..e9b906b36 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -122,3 +122,20 @@ It turns out that the core routing forms of Qi fulfill the definition of @hyperl So evidently, flows are just @hyperlink["https://www.sciencedirect.com/science/article/pii/S1571066106001666/pdf"]{monoids in suitable subcategories of bifunctors} (what's the problem?), or, in another way of looking at it, @hyperlink["https://bentnib.org/arrows.pdf"]{enriched Freyd categories}. Therefore, any theoretical results about arrows should generally apply to Qi as well (but not necessarily, since Qi is not @emph{just} arrows). + +@section{It's Languages All the Way Down} + +Qi is a language implemented on top of another language, Racket, by means of a macro called @racket[flow]. All of the other macros that serve as Qi's @seclink["Embedding_a_Hosted_Language"]{embedding} into Racket, such as (the Racket macros) @racket[~>] and @racket[switch], expand to a use of @racket[flow]. + +@racket[flow] accepts Qi syntax and (like any @tech/reference{macro}) produces Racket syntax. It does this in two stages: + +@itemlist[#:style 'ordered + @item{Expansion, where the Qi source expression is translated to a small core language (Core Qi).} + @item{Compilation, where the Core Qi expression is optimized and then translated into Racket.} +] + +All of this happens at @seclink["phases" #:doc '(lib "scribblings/guide/guide.scrbl")]{compile time}, and consequently, the generated Racket code is then itself @seclink["expansion" #:doc '(lib "scribblings/reference/reference.scrbl")]{expanded} to a @seclink["fully-expanded" #:doc '(lib "scribblings/reference/reference.scrbl")]{small core language} and then @tech/reference{compiled} to @seclink["JIT" #:doc '(lib "scribblings/guide/guide.scrbl")]{bytecode} for evaluation in the runtime environment, as usual. + +Thus, Qi is a special kind of @seclink["Hosted_Languages"]{hosted language}, one that happens to have the same architecture as the host language, Racket, in terms of having distinct expansion and compilation steps. This gives it a lot of flexibility in its implementation, including allowing much of its surface syntax to be implemented as @seclink["Qi_Macros"]{Qi macros} (for instance, Qi's @racket[switch] expands to a use of Qi's @racket[if] just as Racket's @racket[cond] expands to a use of Racket's @racket[if]), allowing it to be naturally macro-extensible by users, and lending it the ability to @seclink["Don_t_Stop_Me_Now"]{perform optimizations on the core language} that allow idiomatic code to be performant. + +This architecture is achieved through the use of @seclink["top" #:indirect? #t #:doc '(lib "syntax-spec-v1/scribblings/main.scrbl")]{Syntax Spec}, following the general approach described in @hyperlink["https://dl.acm.org/doi/abs/10.1145/3428297"]{Macros for Domain-Specific Languages (Ballantyne et. al.)}. From 63580d7bb4a5d74a5a0f9102dae531b3690d803b Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Sat, 20 Jan 2024 18:34:55 -0700 Subject: [PATCH 13/26] doc: Add Qi logo --- qi-doc/scribblings/assets/img/logo.svg | 7 +++++++ qi-doc/scribblings/qi.scrbl | 7 +++++++ 2 files changed, 14 insertions(+) create mode 100644 qi-doc/scribblings/assets/img/logo.svg diff --git a/qi-doc/scribblings/assets/img/logo.svg b/qi-doc/scribblings/assets/img/logo.svg new file mode 100644 index 000000000..6eb66898e --- /dev/null +++ b/qi-doc/scribblings/assets/img/logo.svg @@ -0,0 +1,7 @@ + + + + + + + diff --git a/qi-doc/scribblings/qi.scrbl b/qi-doc/scribblings/qi.scrbl index de3f48408..b6bdc65b8 100644 --- a/qi-doc/scribblings/qi.scrbl +++ b/qi-doc/scribblings/qi.scrbl @@ -9,6 +9,13 @@ An embeddable, general-purpose language to allow convenient framing of programming logic in terms of functional @tech{flows}. A flow is a function from inputs to outputs, and Qi provides compact notation for describing complex flows. +@; Modified from Maciej Barc's req package +@(let ([logo-path + "scribblings/assets/img/logo.svg"]) + (if (file-exists? logo-path) + (centered (image logo-path #:scale 0.7)) + (printf "[WARNING] No ~a file found!~%" logo-path))) + Tired of writing long functional pipelines with nested syntax like this? @racketblock[(map _f (filter _g (vector->list _my-awesome-data)))] Then Qi is for you! From 949c0069a0d0c4db9bf08825e4105f92a3cfa115 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Sat, 20 Jan 2024 18:56:21 -0700 Subject: [PATCH 14/26] doc: another crosslink --- qi-doc/scribblings/field-guide.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 714b8253e..76e4ead3a 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -202,7 +202,7 @@ Methodical use of @racket[gen] together with the @seclink["Using_a_Probe"]{probe ; in: lambda } -@bold{Meaning}: The expander (either the Racket or Qi expander) received syntax, in this case simply "lambda", that it considers to be invalid. Note that if it received something it didn't know anything about, it would say "undefined" rather than "bad syntax." Bad syntax indicates known syntax used in an incorrect way. +@bold{Meaning}: The expander (@seclink["It_s_Languages_All_the_Way_Down"]{either the Racket or Qi expander}) received syntax, in this case simply "lambda", that it considers to be invalid. Note that if it received something it didn't know anything about, it would say "undefined" rather than "bad syntax." Bad syntax indicates known syntax used in an incorrect way. @bold{Common example}: A Racket expression has not been properly escaped within a Qi context. For instance, @racket[(☯ (lambda (x) x))] is invalid because the wrapped expression is Racket rather than Qi. To fix this, use @racket[esc], as in @racket[(☯ (esc (lambda (x) x)))]. From 475b9312f9dac278c5aa1efee36dd43827b8125d Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Sat, 20 Jan 2024 21:57:07 -0700 Subject: [PATCH 15/26] doc: crosslinks --- qi-doc/scribblings/intro.scrbl | 2 +- qi-doc/scribblings/using-qi.scrbl | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/qi-doc/scribblings/intro.scrbl b/qi-doc/scribblings/intro.scrbl index 86adb119c..eb3b4a356 100644 --- a/qi-doc/scribblings/intro.scrbl +++ b/qi-doc/scribblings/intro.scrbl @@ -73,7 +73,7 @@ For macros, we cannot use them naively as @tech{flows} because macros expect all The threading library also provides numerous shorthands for common cases, many of which don't have equivalents in Qi -- if you'd like to have these, please @hyperlink["https://github.com/drym-org/qi/issues/"]{create an issue} on the source repo to register your interest. -Finally, by virtue of having an optimizing compiler, Qi also offers performance benefits in some cases, including for use of sequences of standard functional operations on lists like @racket[map] and @racket[filter], which in Qi @seclink["Don_t_Stop_Me_Now"]{avoid constructing intermediate representations} along the way to generating the final result. +Finally, by virtue of having an @seclink["It_s_Languages_All_the_Way_Down"]{optimizing compiler}, Qi also offers performance benefits in some cases, including for use of sequences of standard functional operations on lists like @racket[map] and @racket[filter], which in Qi @seclink["Don_t_Stop_Me_Now"]{avoid constructing intermediate representations} along the way to generating the final result. @close-eval[eval-for-docs] @(set! eval-for-docs #f) diff --git a/qi-doc/scribblings/using-qi.scrbl b/qi-doc/scribblings/using-qi.scrbl index fe4981add..baa6384cc 100644 --- a/qi-doc/scribblings/using-qi.scrbl +++ b/qi-doc/scribblings/using-qi.scrbl @@ -190,7 +190,7 @@ This succinctness is possible because Qi reaps the twin benefits of (1) working @section{Don't Stop Me Now} -When you're interested in functionally transforming lists using operations like @racket[map], @racket[filter], @racket[foldl] and @racket[foldr], Qi is a good choice because its optimizing compiler eliminates intermediate representations that would ordinarily be constructed in computing the result of such a sequence, resulting in significant performance gains in some cases. +When you're interested in functionally transforming lists using operations like @racket[map], @racket[filter], @racket[foldl] and @racket[foldr], Qi is a good choice because its @seclink["It_s_Languages_All_the_Way_Down"]{optimizing compiler} eliminates intermediate representations that would ordinarily be constructed in computing the result of such a sequence, resulting in significant performance gains in some cases. For example, consider the Racket function: From ae8ff30fd85184bd21961fcfde84be3adda97bc7 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Wed, 24 Jan 2024 19:08:39 -0700 Subject: [PATCH 16/26] Address Ben's review comments --- qi-doc/scribblings/field-guide.scrbl | 4 ++-- qi-doc/scribblings/interface.scrbl | 2 ++ qi-doc/scribblings/principles.scrbl | 2 +- qi-doc/scribblings/qi.scrbl | 10 +++++----- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 76e4ead3a..63e54d3b4 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -35,7 +35,7 @@ Qi encourages a style that avoids "accidental" effects. In functional programming, "effects" refer to anything that the function does that is not captured in its inputs and outputs. This could include things like printing to the screen, writing to a file, or mutating a global variable. -A @tech{flow} should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. +A @tech{flow} should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with ordinary inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. For example, say that we wish to perform a simple numeric transformation on an input number, but also print the intermediate values to the screen. We might do it this way: @@ -403,7 +403,7 @@ Yet, either implementation produces the same output: @racket[(list 1 9 25)]. So, to reiterate, while the output of Qi flows will be the same as the output of equivalent Racket expressions, they may nevertheless exhibit a different order of effects. -If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, define your flow using @racket[esc] (although this would lose any Qi compiler optimizations). +If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, @seclink["Using_Racket_to_Define_Flows"]{define your flow in Racket} by using @racket[esc]. @section{Effectively Using Feedback Loops} diff --git a/qi-doc/scribblings/interface.scrbl b/qi-doc/scribblings/interface.scrbl index 768349ec5..8d9d06532 100644 --- a/qi-doc/scribblings/interface.scrbl +++ b/qi-doc/scribblings/interface.scrbl @@ -352,6 +352,8 @@ The first and most common way is to simply wrap the expression with a @racket[ge The second way is if you want to describe a @tech{flow} using the host language instead of Qi. In this case, use the @racket[esc] form. The wrapped expression in this case @emph{must} evaluate to a function, since functions are the only values describable in the host language that can be treated as flows. Note that use of @racket[esc] is unnecessary for function identifiers since these are @seclink["Identifiers"]{usable as flows directly}, and these can even be @seclink["Templates_and_Partial_Application"]{partially applied using standard application syntax}, optionally with @racket[_] and @racket[___] to indicate argument placement. But you may still need @racket[esc] in the specific case where the identifier collides with a Qi form. +Note that if you use @racket[esc] to define your flow, the @seclink["It_s_Languages_All_the_Way_Down"]{Qi compiler} will not attempt to optimize it. If such a flow occurs within a larger flow, other parts of that containing flow would still be optimized as usual, but as escaped forms are generally not considered in optimizations, there may be few applicable nonlocal optimizations (i.e. those over more than one flow, including for @seclink["Phrases"]{phrases}) involving the escaped flow. + @examples[ #:eval eval-for-docs (define-flow add-two diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index e9b906b36..3748ade57 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -127,7 +127,7 @@ Therefore, any theoretical results about arrows should generally apply to Qi as Qi is a language implemented on top of another language, Racket, by means of a macro called @racket[flow]. All of the other macros that serve as Qi's @seclink["Embedding_a_Hosted_Language"]{embedding} into Racket, such as (the Racket macros) @racket[~>] and @racket[switch], expand to a use of @racket[flow]. -@racket[flow] accepts Qi syntax and (like any @tech/reference{macro}) produces Racket syntax. It does this in two stages: +The @racket[flow] form accepts Qi syntax and (like any @tech/reference{macro}) produces Racket syntax. It does this in two stages: @itemlist[#:style 'ordered @item{Expansion, where the Qi source expression is translated to a small core language (Core Qi).} diff --git a/qi-doc/scribblings/qi.scrbl b/qi-doc/scribblings/qi.scrbl index b6bdc65b8..dc7384f32 100644 --- a/qi-doc/scribblings/qi.scrbl +++ b/qi-doc/scribblings/qi.scrbl @@ -1,5 +1,6 @@ #lang scribble/manual @require[scribble-abbrevs/manual + racket/runtime-path @for-label[qi racket]] @@ -10,11 +11,10 @@ An embeddable, general-purpose language to allow convenient framing of programming logic in terms of functional @tech{flows}. A flow is a function from inputs to outputs, and Qi provides compact notation for describing complex flows. @; Modified from Maciej Barc's req package -@(let ([logo-path - "scribblings/assets/img/logo.svg"]) - (if (file-exists? logo-path) - (centered (image logo-path #:scale 0.7)) - (printf "[WARNING] No ~a file found!~%" logo-path))) +@(define-runtime-path logo-path "assets/img/logo.svg") +@(if (file-exists? logo-path) + (centered (image logo-path #:scale 0.7)) + (printf "[WARNING] No ~a file found!~%" logo-path)) Tired of writing long functional pipelines with nested syntax like this? @racketblock[(map _f (filter _g (vector->list _my-awesome-data)))] From df8e54a6331f886c0221dc34e43527284c35416d Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 25 Jan 2024 01:34:06 -0700 Subject: [PATCH 17/26] address mb review comment --- qi-doc/scribblings/field-guide.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 63e54d3b4..7df25f9ec 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -401,7 +401,7 @@ The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr) Yet, either implementation produces the same output: @racket[(list 1 9 25)]. -So, to reiterate, while the output of Qi flows will be the same as the output of equivalent Racket expressions, they may nevertheless exhibit a different order of effects. +So, to reiterate, while the output of @emph{pure} Qi flows will be the same as the output of equivalent Racket expressions, the behavior of flows with implicit effects may differ, from exhibiting a different order of effects to even producing different output, in the case where the output is dependent on such effects. If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, @seclink["Using_Racket_to_Define_Flows"]{define your flow in Racket} by using @racket[esc]. From ec44d469cc80996f194fbcaf8057b5c47adadb39 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 25 Jan 2024 16:18:14 -0700 Subject: [PATCH 18/26] doc: a correction --- qi-doc/scribblings/field-guide.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 7df25f9ec..3c3c1e54e 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -401,7 +401,7 @@ The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr) Yet, either implementation produces the same output: @racket[(list 1 9 25)]. -So, to reiterate, while the output of @emph{pure} Qi flows will be the same as the output of equivalent Racket expressions, the behavior of flows with implicit effects may differ, from exhibiting a different order of effects to even producing different output, in the case where the output is dependent on such effects. +So, to reiterate, while the behavior of @emph{pure} Qi flows will be the same as that of equivalent Racket expressions, effectful flows may exhibit a different order of effects. In the case where the output of such effectful flows is dependent on those effects (such as relying on a mutable global variable), these flows could even produce different output than otherwise equivalent (from the perspective of inputs and outputs, disregarding effects) Racket or even Qi code. If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, @seclink["Using_Racket_to_Define_Flows"]{define your flow in Racket} by using @racket[esc]. From 826256d6cc1e20a6860ebcf68b87f40e8865ad27 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 25 Jan 2024 19:14:46 -0700 Subject: [PATCH 19/26] doc: avoid confusion around "define"; minor edit re: effects in Qi --- qi-doc/scribblings/field-guide.scrbl | 4 ++-- qi-doc/scribblings/interface.scrbl | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 3c3c1e54e..5f468e5d1 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -401,9 +401,9 @@ The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr) Yet, either implementation produces the same output: @racket[(list 1 9 25)]. -So, to reiterate, while the behavior of @emph{pure} Qi flows will be the same as that of equivalent Racket expressions, effectful flows may exhibit a different order of effects. In the case where the output of such effectful flows is dependent on those effects (such as relying on a mutable global variable), these flows could even produce different output than otherwise equivalent (from the perspective of inputs and outputs, disregarding effects) Racket or even Qi code. +So, to reiterate, while the behavior of @emph{pure} Qi flows will be the same as that of equivalent Racket expressions, effectful flows may exhibit a different order of effects. In the case where the output of such effectful flows is dependent on those effects (such as relying on a mutable global variable), these flows could even produce different output than otherwise equivalent (from the perspective of inputs and outputs, disregarding effects) Racket code. -If you'd like to ensure a particular order of effects, use @racket[effect] at the appropriate points in your flow. If you'd like to use Racket's order of effects, @seclink["Using_Racket_to_Define_Flows"]{define your flow in Racket} by using @racket[esc]. +If you'd like to use Racket's order of effects in any flow, @seclink["Using_Racket_to_Define_Flows"]{write the flow in Racket} by using a wrapping @racket[esc]. @section{Effectively Using Feedback Loops} diff --git a/qi-doc/scribblings/interface.scrbl b/qi-doc/scribblings/interface.scrbl index 8d9d06532..0c7858afb 100644 --- a/qi-doc/scribblings/interface.scrbl +++ b/qi-doc/scribblings/interface.scrbl @@ -336,7 +336,7 @@ The advantage of using these over the general-purpose @racket[define] form is th @section{Using the Host Language from Qi} -Arbitrary native (e.g. Racket) expressions can be used in @tech{flows} in one of two core ways. This section describes these two ways and also discusses other considerations regarding use of the host language alongside Qi. +Arbitrary host language (e.g. Racket) expressions can be used in @tech{flows} in one of two core ways. This section describes these two ways and also discusses other considerations regarding use of the host language alongside Qi. @subsection{Using Racket Values in Qi Flows} @@ -352,13 +352,14 @@ The first and most common way is to simply wrap the expression with a @racket[ge The second way is if you want to describe a @tech{flow} using the host language instead of Qi. In this case, use the @racket[esc] form. The wrapped expression in this case @emph{must} evaluate to a function, since functions are the only values describable in the host language that can be treated as flows. Note that use of @racket[esc] is unnecessary for function identifiers since these are @seclink["Identifiers"]{usable as flows directly}, and these can even be @seclink["Templates_and_Partial_Application"]{partially applied using standard application syntax}, optionally with @racket[_] and @racket[___] to indicate argument placement. But you may still need @racket[esc] in the specific case where the identifier collides with a Qi form. -Note that if you use @racket[esc] to define your flow, the @seclink["It_s_Languages_All_the_Way_Down"]{Qi compiler} will not attempt to optimize it. If such a flow occurs within a larger flow, other parts of that containing flow would still be optimized as usual, but as escaped forms are generally not considered in optimizations, there may be few applicable nonlocal optimizations (i.e. those over more than one flow, including for @seclink["Phrases"]{phrases}) involving the escaped flow. +Finally, bear in mind that if you use @racket[esc], the @seclink["It_s_Languages_All_the_Way_Down"]{Qi compiler} will not attempt to optimize the resulting flow. If such a flow occurs within a larger flow, other parts of that containing flow would still be optimized as usual, but as escaped forms are generally not considered in optimizations, there may be few applicable nonlocal optimizations (i.e. those over more than one flow, including for @seclink["Phrases"]{phrases}) involving the escaped flow. @examples[ #:eval eval-for-docs (define-flow add-two (esc (λ (a b) (+ a b)))) (~> (3 5) add-two) + (~> (3) sqr (esc (λ (x) (+ x 5)))) ] @subsection{Using Racket Macros as Flows} From 97f9568f9ddc6b62e360d08f7c816cc510187390 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Fri, 26 Jan 2024 00:24:52 -0700 Subject: [PATCH 20/26] doc: separate effects from other computations Decouple the discussion about separating effects from other computations from the discussion about order of effects. --- qi-doc/scribblings/field-guide.scrbl | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 5f468e5d1..6c0097072 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -29,15 +29,17 @@ Decompose your @tech{flow} into its smallest components, and name each so that t A journeyman of one's craft -- a woodworker, electrician, or a plumber, say -- always goes to work with a trusty toolbox that contains the tools of the trade, some perhaps even of their own design. An electrician, for instance, may have a voltage tester, a multimeter, and a continuity tester in her toolbox. Although these are "debugging" tools, they aren't just for identifying bugs -- by providing rapid feedback, they enable her to explore and find creative solutions quickly and reliably. It's the same with Qi. Learn to use the @seclink["Debugging"]{debugging tools}, and use them often. -@subsection{Be Intentional About Effects} +@subsection{Separate Effects from Other Computations} -Qi encourages a style that avoids "accidental" effects. +In functional programming, "effects" refer to anything a function does that is not captured in its inputs and outputs. This could include things like printing to the screen, writing to a file, or mutating a global variable. -In functional programming, "effects" refer to anything that the function does that is not captured in its inputs and outputs. This could include things like printing to the screen, writing to a file, or mutating a global variable. +In general, pure functions (that is, functions free of such effects) are easier to understand and easier to reuse, and favoring their use is considered good functional style. But of course, it's necessary for your code to actually do things besides compute values, too! There are many ways in which you might combine effects and pure functions, from mixing them freely, as you might in Racket, to extracting them completely using monads, as you might in Haskell. Qi encourages using pure functions side by side with what we could call "pure effects." -A @tech{flow} should either be pure (that is, free of such side effects), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with ordinary inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and perform the effect explicitly via the @racket[effect] form, or otherwise escape from Qi using something like @racket[esc] (note that @seclink["Identifiers"]{function identifiers} used in a flow context are implicitly @racket[esc]aped) in order to perform the effect. This will ensure that there are no surprises with regard to @seclink["Order_of_Effects"]{order of effects}. +If you have a function with ordinary inputs and outputs that also performs an effect, then, to adopt this style, decouple the effect from the rest of the function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and then invoke it via an explicit use of the @racket[effect] form, thus neatly separating the functional computation from the effect. -For example, say that we wish to perform a simple numeric transformation on an input number, but also print the intermediate values to the screen. We might do it this way: +Doing it this way encourages smaller, well-scoped functions that do one thing, and which serve as excellent building blocks from which to compose large and complex programs. In contrast, larger, effectful functions make poor building blocks and are difficult to compose. + +To illustrate, say that we wish to perform a simple numeric transformation on an input number, but also print the intermediate values to the screen. We might do it this way: @examples[ #:eval eval-for-docs @@ -53,7 +55,9 @@ For example, say that we wish to perform a simple numeric transformation on an i (~> (3) my-square my-add1) ] -This is considered poor style since we've mixed pure functions with implicit effects. Instead, following the above guideline, we would write it this way: +This is considered poor style since we've mixed pure functions with implicit effects. It makes these functions less portable since we might find use for such computations in other settings where we might prefer to avoid the side effect, or perhaps perform a different effect like writing the value to a network port. With the functions written this way, we would be encouraged to write similar functions in these different settings, exhibiting the other effects we might desire there, and duplicating the core logic. + +Instead, following the above guideline, we would write it this way: @examples[ #:eval eval-for-docs @@ -61,7 +65,7 @@ This is considered poor style since we've mixed pure functions with implicit eff (~> (3) (ε displayln sqr) (ε displayln add1)) ] -This uses the pure functions @racket[sqr] and @racket[add1], extracting the effectful @racket[displayln] as an explicit @racket[effect]. +This uses the pure functions @racket[sqr] and @racket[add1], extracting the effectful @racket[displayln] as an explicit @racket[effect]. If we wanted to have other effects, we could simply indicate different effects here and reuse the same underlying pure functions. @section{Debugging} @@ -393,7 +397,7 @@ So in general, use mutable values with caution. Such values can be useful as sid @subsubsection{Order of Effects} - Qi @tech{flows} may exhibit a different order of effects (in the @seclink["Be_Intentional_About_Effects"]{functional programming sense}) than equivalent Racket functions. + Qi @tech{flows} may exhibit a different order of effects (in the @seclink["Separate_Effects_from_Other_Computations"]{functional programming sense}) than equivalent Racket functions. Consider the Racket expression: @racket[(map sqr (filter odd? (list 1 2 3 4 5)))]. As this invokes @racket[odd?] on all of the elements of the input list, followed by @racket[sqr] on all of the elements of the intermediate list, if we imagine that @racket[odd?] and @racket[sqr] print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence @racket[1,2,3,4,5,1,3,5]. From 7471bd1087956d43ce615f64220754e035352093 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Fri, 26 Jan 2024 02:42:33 -0700 Subject: [PATCH 21/26] doc: ask readers to report inscrutable errors --- qi-doc/scribblings/field-guide.scrbl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 6c0097072..5220e8648 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -159,6 +159,8 @@ Methodical use of @racket[gen] together with the @seclink["Using_a_Probe"]{probe @subsection{Common Errors and What They Mean} +Qi aims to produce good error messages that convey what the problem is and clearly imply a remedy. For various reasons, it may not always be possible to provide such a clear message. This section documents known errors of this kind, and suggests possible causes and remedies. If you encounter an inscrutable error, please consider @hyperlink["https://github.com/drym-org/qi/issues/"]{reporting it}. If the error cannot be improved, then it will be documented here. + @subsubsection{Expected Number of Values Not Received} @codeblock{ From 0a59e2432742850b68221cfbb0c5bd8b32d7bca2 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 29 Jan 2024 21:56:46 -0700 Subject: [PATCH 22/26] doc: note another common error re: threading direction --- qi-doc/scribblings/field-guide.scrbl | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 5220e8648..53a1726c1 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -298,6 +298,18 @@ Qi aims to produce good error messages that convey what the problem is and clear @bold{Common example}: Attempting to use a Qi macro in one module without @racketlink[provide]{providing} it from the module where it is defined -- note that Qi macros must be provided as @racket[(provide (for-space qi mac))]. See @secref["Using_Macros" #:doc '(lib "qi/scribblings/qi.scrbl")] for more on this. +@subsubsection{Contract Violation} + +@codeblock{ +; map: contract violation +; expected: procedure? +; given: '(1 2 3) +} + +@bold{Meaning}: The interpreter attempted to apply a function to arguments but found that an argument was not of the expected type. + +@bold{Common example}: Using a nested flow (such as a @racket[tee] junction or an @racket[effect]) within a right-threading flow and assuming that the input arguments would be passed on the right. At the moment, Qi does not propagate the threading direction to nested clauses. You could either use a fresh right threading form or indicate the argument positions explicitly in the nested flow using an @seclink["Templates_and_Partial_Application"]{argument template}. + @subsubsection{Compose: Contract Violation} @codeblock{ From 530ca6064f4ca9c854c1ad6b35ddd48e94c3fc61 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Wed, 21 Feb 2024 17:35:53 -0800 Subject: [PATCH 23/26] doc: shorten a sentence and attempt to make it clearer --- qi-doc/scribblings/interface.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/interface.scrbl b/qi-doc/scribblings/interface.scrbl index 0c7858afb..59a6d8632 100644 --- a/qi-doc/scribblings/interface.scrbl +++ b/qi-doc/scribblings/interface.scrbl @@ -352,7 +352,7 @@ The first and most common way is to simply wrap the expression with a @racket[ge The second way is if you want to describe a @tech{flow} using the host language instead of Qi. In this case, use the @racket[esc] form. The wrapped expression in this case @emph{must} evaluate to a function, since functions are the only values describable in the host language that can be treated as flows. Note that use of @racket[esc] is unnecessary for function identifiers since these are @seclink["Identifiers"]{usable as flows directly}, and these can even be @seclink["Templates_and_Partial_Application"]{partially applied using standard application syntax}, optionally with @racket[_] and @racket[___] to indicate argument placement. But you may still need @racket[esc] in the specific case where the identifier collides with a Qi form. -Finally, bear in mind that if you use @racket[esc], the @seclink["It_s_Languages_All_the_Way_Down"]{Qi compiler} will not attempt to optimize the resulting flow. If such a flow occurs within a larger flow, other parts of that containing flow would still be optimized as usual, but as escaped forms are generally not considered in optimizations, there may be few applicable nonlocal optimizations (i.e. those over more than one flow, including for @seclink["Phrases"]{phrases}) involving the escaped flow. +Finally, bear in mind that if you use @racket[esc], the @seclink["It_s_Languages_All_the_Way_Down"]{Qi compiler} will not attempt to optimize the resulting flow. Of course, if such a flow occurs within a larger flow, other parts of that containing flow would still be optimized as usual. Yet, as escaped forms are generally not considered in optimizations, the presence of such forms may make it less likely that there would be applicable nonlocal (i.e. involving more than one flow, like @seclink["Phrases"]{phrases}) optimizations. @examples[ #:eval eval-for-docs From 52fc4d6566a33664745727ac59329de4c4197aef Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 12:57:07 -0700 Subject: [PATCH 24/26] doc: Some reordering and elaboration of principles --- qi-doc/scribblings/principles.scrbl | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 3748ade57..a3fab2d43 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -27,14 +27,18 @@ The Qi language allows you to describe and use flows in your code. -@section{Values, Paths and Flows} +@section{Values and Flows} @tech{Flows} accept inputs and produce outputs -- they are functions. The things that flow -- the inputs and outputs -- are @emph{values}. Yet, values do not actually "move" through a flow, since a flow does not mutate them. The flow simply produces new values that are related to the inputs by a computation. - Every flow is made up of components that are themselves flows. Thus, each of these components is a relationship between an input set of values and an output set of values, so that at every level, flows produce sequences of sets of values beginning with the inputs and ending with the outputs, with each set related to the preceding one by a computation, and again, no real "motion" of values at all. There may be many such distinct @deftech{paths} over flow components that could be traced (borrowing the term "path" as used in graph theory in this sense), and we may imagine values to flow along these paths. + Every flow is made up of components that are themselves flows. Thus, each of these components is a relationship between an input set of values and an output set of values, so that at every level, flows produce sequences of sets of values beginning with the inputs and ending with the outputs, with each set related to the preceding one by a computation, and again, no real "motion" of values at all. So indeed, when we say that values "flow," there is nothing in fact that truly flows, and it is merely a convenient metaphor. +@section{Flows as Graphs} + + A flow could also be considered an @hyperlink["https://en.wikipedia.org/wiki/Directed_acyclic_graph"]{acyclic graph}, with its component flows as nodes, and a directed edge connecting two flows if an output of one is used as an input of the other. There may be many distinct @deftech{paths} that could be traced over this graph, and we may imagine values to flow along these paths at runtime (although of course, @seclink["Values_and_Flows"]{there is nothing that flows}). At each point in the flow (in this spatial sense), there is a certain number of values present, depending on the runtime inputs. We refer to this number as the @deftech{arity} or the @deftech{volume} of the flow at that point. Volume is a runtime concept since it depends on the actual inputs provided to the flow, although there may be cases where it could be determined at compile time. + @section{Values are Not Collections} The things that flow are values. Individual values may happen to be collections such as lists, but the values that are flowing are not, together, a collection of any kind. From bce402b20d569374ef1743a2b0d3579c91a4e79d Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 13:01:23 -0700 Subject: [PATCH 25/26] doc: Using Qi as a dependency --- qi-doc/scribblings/intro.scrbl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/qi-doc/scribblings/intro.scrbl b/qi-doc/scribblings/intro.scrbl index eb3b4a356..7a490238f 100644 --- a/qi-doc/scribblings/intro.scrbl +++ b/qi-doc/scribblings/intro.scrbl @@ -63,6 +63,12 @@ Qi is a hosted language on the @hyperlink["https://racket-lang.org/"]{Racket pla Since some of the forms use and favor unicode characters (while also providing plain-English aliases), see @secref["Flowing_with_the_Flow"] for tips on entering these characters. Otherwise, if you're all set, head on over to the @seclink["Tutorial"]{tutorial}. +@section{Using Qi as a Dependency} + + Qi follows the @hyperlink["https://countvajhula.com/2022/02/22/how-to-organize-your-racket-library/"]{composable package organization scheme}, so that you typically only need to depend on @code{qi-lib} in your @seclink["metadata" #:doc '(lib "pkg/scribblings/pkg.scrbl")]{application or library}. The @code{qi-lib} package entails just those dependencies used in the Qi language itself, rather than those used in tests, benchmarking, documentation, etc. All of those dependencies are encapsulated in separate packages such as @code{qi-test}, @code{qi-doc}, @code{qi-sdk}, and more. This ensures that using Qi as a dependency contributes minimal overhead to your build times. + + Additionally, Qi itself uses few and carefully benchmarked dependencies, so that the load-time overhead of @racket[(require qi)] is minimal. + @section{Relationship to the Threading Macro} The usual threading macro in @seclink["top" #:indirect? #t #:doc '(lib "scribblings/threading.scrbl")]{Threading Macros} is a purely syntactic transformation that does not make any assumptions about the expressions being threaded through, so that it works out of the box for threading values through both functions as well as macros. On the other hand, Qi is primarily oriented around @emph{functions}, and @tech{flows} are expected to be @seclink["What_is_a_Flow_"]{function-valued}. Threading values through macros using Qi requires special handling. From 30e8e2ac0cd197460839855802110aac6bf6d62f Mon Sep 17 00:00:00 2001 From: Siddhartha Kasivajhula Date: Thu, 14 Mar 2024 14:20:07 -0700 Subject: [PATCH 26/26] doc: fix wording (cr) Co-authored-by: D. Ben Knoble --- qi-doc/scribblings/principles.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index a3fab2d43..09afcdde2 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -37,7 +37,7 @@ @section{Flows as Graphs} - A flow could also be considered an @hyperlink["https://en.wikipedia.org/wiki/Directed_acyclic_graph"]{acyclic graph}, with its component flows as nodes, and a directed edge connecting two flows if an output of one is used as an input of the other. There may be many distinct @deftech{paths} that could be traced over this graph, and we may imagine values to flow along these paths at runtime (although of course, @seclink["Values_and_Flows"]{there is nothing that flows}). At each point in the flow (in this spatial sense), there is a certain number of values present, depending on the runtime inputs. We refer to this number as the @deftech{arity} or the @deftech{volume} of the flow at that point. Volume is a runtime concept since it depends on the actual inputs provided to the flow, although there may be cases where it could be determined at compile time. + A flow could also be considered an @hyperlink["https://en.wikipedia.org/wiki/Directed_acyclic_graph"]{acyclic graph}, with its component flows as nodes, and a directed edge connecting two flows if an output of one is used as an input of the other. There may be many distinct @deftech{paths} that could be traced over this graph, and we may imagine values to flow along these paths at runtime (although of course, @seclink["Values_and_Flows"]{there is nothing that flows}). At each point in the flow (in this spatial sense), there are a certain number of values present, depending on the runtime inputs. We refer to this number as the @deftech{arity} or the @deftech{volume} of the flow at that point. Volume is a runtime concept since it depends on the actual inputs provided to the flow, although there may be cases where it could be determined at compile time. @section{Values are Not Collections}