Tutorial Maker Widget Revamp

[nbriz]

@Cthomp7 wanted to make an issue to track some of the bigger picture plans && discussions around this.

one tiny thing that came to mind this morning was that since we're going to create a widget (for the sake of opening the popup && relaying messages) we should probably also place the pop up folder (with all it's files) within the widget folder (rather than in core).

[nbriz]

Hey @Cthomp7

here's some updates on the popup/iframe "patterns" i've been using on the convo-maker. like we discussed in our meeting, my thinking is that these could be standards we set for all of these meta-tool type widgets (widgets used to work on netnet itself) like the convo-maker, dem-maker && tutorial-maker. this "spec" is still in developmen, so if u have critiques let me know. once we both feel solid about it after working on these widgets I'll put all this into the new docs.

meta-widgets are popups

i think we make the trend u started the requirement for all these meta widgets, if it's a widget one uses to work on netnet itself, then we make it a pop up. in these instances a widget's directory structure should look like this:

• popups (directory with all the popups code, essentially a mini-website, it's plural b/c there may be more than one in there, the convo-maker has two)
• index.js (the main widget file)
• convo.js (the widget's convo file, optional)
• styles.css (since the actual widget has no interface i don't think this is necessary anymore?)

i think the primary way of accessing these widgets will be to search for them in the search bar, once u click the search result the popup should open up (but the widget itself remains "hidden"). if u close the popup u should be able to re-open it when u search for it again in the search bar. In order for all that to work smoothing, i'd suggest the following:

here's an example index.js for a meta-widget

/* global Widget, nn */
class ExampleWidget extends Widget {
  constructor (opts) {
    super(opts)

    this.key = 'example-widget'
    this.listed = true // so that it shows up in the search bar
    this.keywords = ['...'] // search terms for search bar indexing
    this.title = 'Example Widget'
    this.hidden = true

    this.popup = null // this could be called whatever we want, we just need a reference to the popup window

    this.on('open', () => { // if we open this widget (ie. through the search bar) automatically open popup
      if (!this.popup) this._openPopup()
    })

    // listen for messages from the popup in case anything needs to be handled
    // within netnet by the widget, for example displaying a dialogue/passage
    // or interacting with other widgets, the netnet editor, etc.
    nn.on('message', e => {
      if (e.origin !== window.location.origin) return // for security
      if (e.data.type === 'example-opened') {
        console.log(e.data.payload) // do something with payload
      }
    })

    // if we quit or refresh the netnet tab, we should also close the popup
    nn.on('beforeunload', () => {
      if (this.popup && !this.popup.closed) this.popup.close()
    })
  }

  _openPopup (type, payload) {
    // create pop up
    const url = 'widgets/example-widget/popups/index.html'
    this.popup = window.open(url, 'example-widget', 'width=1000,height=620')
    // keep an eye on the pop up to see if it closed
    this.popupWatcher = setInterval(() => {
      if (this.popup && this.popup.closed) {
        clearInterval(this.popupWatcher)
        this.popup = null
      }
    }, 500)
  }

  // in case we need to send any messages back to the popup
  _messagePopup (type, payload) {
    if (!this.popup) return
    this.popup.postMessage({ type, payload }, window.origin)
  }
}

NOTE: I mentioned in my email that in order for that to work the core Widget's event system needs to be able to fire/emit the "open" event whether or not the widget is "hidden", that's the reason i had to add this line:
https://github.com/nbriz/netnet.studio/blob/e5662521407a1a54c48e7779ba1ed042a4d035c9/www/widgets/index.js#L241

window messaging protocol

When we create a "message" listener on the window (ie. nn.on('message', ...)) we're not just listening for messages from our popup, this listens for any messages coming from other windows, be those iframes like the rendered output from the netnet editor code (that iframe sends back console errors so netnet can display them) or messages coming from any popup (whether or not t belongs to this widget). so in order to avoid conflicts/confusion i propose we create the following convention:

all messages sent to/from popups should always be an object which contain at least a type property, as well as an optional payload property && nothing else.

• type this should be a string which is prefixed with the widget's name (or some abbreviation thereof), so u might do type: 'tutmkr-opened' and type: tutmkr-update or something like that.
• payload if the message has any extra data to send, this should be stored in the payload property (this could be any data including arrays && objects if lots of stuff needs to be sent over)

in the template i put together above u'll notice that the listener is checking only/specifically for events pertaining to this widget && ignoring any others that might get picked up from other widgets.

nn.on('message', e => {
  if (e.origin !== window.location.origin) return // for security
  if (e.data.type === 'example-opened') {
    console.log(e.data.payload) // do something with payload
  }
})

also, the method for messaging the popup assumes "type" && "payload" values:

_messagePopup (type, payload) {
  if (!this.popup) return
  this.popup.postMessage({ type, payload }, window.origin)
}

which could be used anywhere in the widget like this:

// inside some other method
this._messagePopup('tutmkr-new-data', { name: 'something', duration: 100 })
```

[nbriz]

hey @Cthomp7 i just sent an email with updates to all the demo related widgets i just made. i mentioned there that i updated the Demo Maker to fit these new popup conventions, so i thought i might mention it here as it could serve as a good reference for the structure of the tutorial maker. https://github.com/netizenorg/netnet.studio/tree/dev/www/widgets/demo-maker

specifically take a look at the index.js which is an implementation of the general convention i described in my last comment above: https://github.com/netizenorg/netnet.studio/blob/dev/www/widgets/demo-maker/index.js

as well as the files for the popup itself:

• index.html which imports CSS/JS from netnet but also from it's own popup
• main.js main JS for the popup, which handles all the communication with the main widget && also makes extensive use of the nn library we talked about to connect all the HTML with it's corresponding JS logic
• modal.js this may or may not be relevant, but this is a little modal i created (in place of generic alert() functions), i did something similar with the convo-maker, so figure it could be useful here too (all the CSS for this is in the styles.css)

[nbriz]

@Cthomp7 as i work on the new data structure something i realized about our new plan (for this to all work online without the need to clone the repo && work locally) is that we don't have a process for converting videos into the appropriate formats.

right now the HyperVideoPlayer will check the users platform support for diff video formats && serve up the one their system can play, this required that each tutorial have at least two versions of their video file, an mp4 && a webm. The way i handled this was i'd record the webm using the Streeeemz recorder && then convert it into an mp4 using ffmpeg, these were the commands i had saved in my notes:

ffmpeg -i input.webm -strict experimental output.mp4

ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -b:a 128k -c:a libopus output.webm

ffmpeg -i input.mov -c:v libx264 -crf 18 -preset slow -c:a aac -b:a 192k output.mp4

now, the reason i would require at least those two versions, is b/c last i checked (few years back when i was first working on these) we'd need at least those two to ensure corss-platform compatability... but maybe this is no longer the case?

so, a few questions we need to answer:

• is it still the case that we need both? or could we get away with one?
• if we could get away with one, is it possible to record in that format on all platforms? (i had an issue recording mp4 on Firefox when i first made the widget, which is why i'd record webm)
• if we do need diff versions, how should we handle converting the recording (be it mp4 or webm) into the other format? considering that we want this to work entirely online (one idea would be to run ffmpeg on our server, but i rather not introduce that load on our backend if we can find an option that works client side)

[Cthomp7]

This is what I was able to find based on AI. I did double-check and without third-party tools recording mp4 on Firefox does not seem possible. ### AI-generated explanation For playback: if you only ship MP4 (H.264 video + AAC audio) you’ll reach essentially all modern browsers and devices (Chrome/Edge/Firefox/Safari on desktop & mobile). WebM playback is broadly supported too, but Safari’s support has been inconsistent across OS/device versions; MP4 remains the safest single target. [BrowserStack](https://www.browserstack.com/guide/html5-codec?utm_source=chatgpt.com)[MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Video_codecs?utm_source=chatgpt.com) For recording in-browser (MediaRecorder): this is where the “one format” dream usually breaks: Chrome/Edge/Safari can record video/mp4 on many platforms now. [Google Groups](https://groups.google.com/a/chromium.org/g/blink-dev/c/YJ1QijNiHeM/m/HWs-wQBfAAAJ?utm_source=chatgpt.com)[MDN Web Docs+1](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/isTypeSupported_static?utm_source=chatgpt.com) Firefox still does not record MP4; it records WebM (VP8/Opus). [Bugzilla](https://bugzilla.mozilla.org/show_bug.cgi?id=1631143&utm_source=chatgpt.com)[MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder?utm_source=chatgpt.com) ➡️ Conclusion: If Firefox is in scope, you’ll end up with WebM from Firefox and MP4 from the Chromium/WebKit family unless you transcode. Use MediaRecorder.isTypeSupported() at runtime to choose the best container per browser. ### To avoid loading the backend there are two options suggested by AI: 1) Client-side transcode (browser) Use FFmpeg compiled to WebAssembly (e.g., ffmpeg.wasm) to turn Firefox’s WebM into MP4 before upload. This avoids backend CPU, but has trade-offs: Pros: Zero server load, privacy-friendly. Cons: ~20–30 MB library download, heavy CPU, slower on low-end devices; needs a Web Worker and progress UX. 2) Asynchronous server-side transcode Upload whatever the browser recorded, then transcode out of band (queue/worker or a media service). This keeps your API fast and pushes CPU to a worker tier you can autoscale. It’s the most reliable at scale and usually cheaper overall than making every client download & run FFmpeg. (If you must avoid your primary backend doing the work, a lightweight queue + worker or a dedicated media microservice still counts as “server-side” without burdening your main API.) Let me know if this helps. It looks like for the most part we can stick to mp4 format however when recording on Firefox we will need to do some extra handling to convert the webm format to mp4.

[nbriz]

thnx @Cthomp7, so yea maybe we stick to mp4 as a single format (we should double check && verify that what AI says is true about it being supported everywhere first), if folks use Chrome they wont' have to worry about much else, but for folks who want to make tutorials on FF (as i likely will ^_^), we'll mention that the webm file the Tutorial Maker creates will need to be converted to mp4 && maybe just provide instructions for the tutorial creator (ie. recommending ffmpeg && pointing to other tools). what do u think of that?

[Cthomp7]

@nbriz I think as long as the documentation is super clear and very easy to access/find I think this is a good solution! I am happy to work on that documentation when the time comes too. Like after download maybe a netnet convo redirects them to the new netnet documentation space Sarah is working on.

[nbriz]

great! let's stick to that then. in my re-worked data structure i'm going to assume an mp4 file && throw an error if one doesn't exist

[nbriz]

ok, here's the new proposed data structure, each tutorial folder should have all it's data stored in a file called tutorial.json, the top level of this json object should have the following keys:

const tutorial = {
  id: 'what-is-code', // id/key this should match the tutorial's folder name && video's filename
  metadata: {}, // metadata (see below)
  widgets: {}, // widgets (see below)
  keyframes: [], // keyframes (see below)
  keylogs: [], // keylogs (see below)
}

the metada object should contain the following:

tutorial.metadata = {
  id: "what-is-code", // should match the root id (duplicated for the learning-guide's convenience)
  title: "What Is Code?", // title of the tutorial
  subtitle: '', // optional sub-title
  duration: 1081.274833,
  author: "Nick Briz",
  authorURL: "https://nickbriz.com",
  jsfile: true, // if this tutorial contains a init.js (see below)
  description: "Whether or not you've ever written code, this tutorial is a good place to start. In addition to orienting you to the studio (this website), in this lesson we'll cover what code is from both a technical perspective but also a conceptual one. Like all human languages, coding languages are multipurpose. In the same way that the English language can be used to write history, laws, love letters or poems, coding languages can be used to express more than just software. Here we'll introduce the particular creative and experimental vector view that we'll be approaching code from throughout these tutorials.",
  keywords: [
    "code",
    "intro",
    "netnet",
    "coding",
    "computers",
    "programming"
  ],
  thumbnails: [
    "images/analyticalengine.jpg",
    "images/punch-card-machine.jpg"
  ]
}

the widgets object is exactly the same as before, it contains a dictionary of widgets, which look like:

tutorial.widgets['ono'] = {
  key: "ono",
  title: "Yoko Ono",
  innerHTML: "<a href=\"tutorials/what-is-code/images/ono.png\" target=\"_blank\"><img style=\"width:100%\" src=\"tutorials/what-is-code/images/ono.png\"></a>\n<br>\n<p>\n  a few of <a href=\"https://en.wikipedia.org/wiki/Yoko_Ono\" target=\"_blank\">Yoko Ono</a>'s <i>instruction</i> pieces.\n</p>",
  type: "Widget"
},

the keylogs property is now an array (instead of it's own file) and contains timecode data (seconds) relative to the start of the video now (instead of milliseconds relative to it's starting keyframe)

const keylogFrameExample = {
  timecode: 392.314305,
  code: "<center>\n  <img src=\"circle.png\" alt=\"an image of a circle\">\n  <p>This is a Circle</p>\n</center>"
}

the keyframes property is now an array (instead of an object) containing objects which look like this:

const keyframeExample = {
  // the timecode in seconds for this keyframe
  timecode: 322.218371,
  // optional keyframe name
  name: "", 
  // object containing the HyperVideoPlayer size/position
  video: {
    width: 521,
    height: 447,
    left: 119,
    bottom: 54,
    zIndex: 118
  },
  // array of widgets in this keyframe && their size/position
  widgets: [
    {
      key: "binary",
      width: 200,
      height: 245,
      right: 191,
      top: 48,
      zIndex: 122
    },
    {
      key: "gates1",
      width: 662,
      height: 389,
      left: 50,
      top: 39,
      zIndex: 124
    }
  ],
  // netitor state for this keyframe (code, scroll position, spotlight values and layout)
  netitor: {
    code: "<style>\n  body::after {\n    content: \"\";\n    position: fixed;\n    left: 15px;\n    top: 15px;\n    width: 50vw;\n    height: 50vw;\n    background-color: white;\n    border-color: black;\n    border-style: solid;\n    border-width: 10px;\n    border-radius: 50%;\n  }\n</style>",
    scrollTo: { x: 0, y: 100 },
    spotlight: ['12', '24-28'],
    layout: "separate-window"
  },
  // the netnet windows size/posiition (when it's in 'welcome' or 'separate-window' layout)
  netnet: {
    right: 43,
    top: 158,
    width: 721,
    height: 445
  }
}

The init.js file is an optional file the tutorial should be able to create, the structure of that file should always look like this:

window.TUTORIAL = {
  init: () => {
    // method which gets called soon as the tutorial loads
  },
  callbacks: {
    463.020014: () => {
      // functions which get called at specific time codes (like keyframes)
    }
  }
}

We could maybe create a pop-out netitor for this (similar to how the Convo Maker creates a pop-out netitor so convo makers can write arbitrary js for convos), but i wouldn't worry about this for now (not that many tutorials use this feature && if someone making a tutorial is leaning into this they could probably create this file in their own code editor for the time being)

[nbriz]

@Cthomp7

so i realized this morning i forgot to mention one thing in our meeting, we discussed how the logic for recording a video would be migrated/refactored from the 'stream-video' widget to the recorder.js file, as well as the logic from netitor-logger (either into that recorder.js file or it's own logger.js file), but forgot to mention that there's also the 'widget-maker', some of that logic will also need to be migrated over into it's own js file for the popup. the widget-maker was created a while back && just used a simple <textarea> for the innerHTML of the widget, but we should probably do the same as we did in the demo-maker && make that field a netitor instance. which means we'll need to import in the index.html like we did here (which will be nice to have in there anyway as we'll eventually just it for the "init.js" jsfile stuff)