Language Models in Programming / archived / read-only

 
  • LLMs today: How good are they?
    • Solving coding challenges
      • Single prompt at its max:
      • Source:
      • |100%
      • New Bard vs GPT-4 (2023-09-20):
        • Using web technologies, create a simple 2D physical simulation of a solar system with one planet. The planet should orbit around the center of the system. - Set up a simple 20 fps game engine using setInterval()! - Set up an area (a
          ) on the page for displaying the simulation! The planet will be displayed using an another
          inside the area. You can avoid displaying the star. - The model of the planet is its P = (x,y) position and V = (vx, vy) speed vector. The center star is fixed at C = (starx, stary) For the sake of simplicity, you do not need to use SI units. Use pixel coordinates directly! - One step of the simulation is defined by the following formulas: x += vx y += vy // The position is modified with the speed. distance =sqrt((starx - x)2 +(stary - y)2) force = G / distance2 // G is the gravitational constant vx += force * (starx -x) / distance vy += force * (stary -y) / distance //The speed is modified with the gravitational force - Set up the P, V and C vectors and G constant at the start of the simulation so that it looks nice. (E.g.: C=(100, 100), P=(200, 100), V=(0, 1), G= 500)
        • GPT-4 solves it at first trial
        • Bard almost solves it, but the code throws errors (2x trial, different errors). After fixing by a new prompt, it works.
    • Working memory (input token count, e.g. 4K, 32K)
      • In the kilobyte range (vs humans?)
      • They need some 'attention' mechanism to work on large codebases
    • Cognitive load they can bear (attention head count (128+) and dimension)
      • Number of distinct commands
        • Junior: 13 fixed + task
      • How far the model must go from its training knowledge
        • E.g. coding in less popular frameworks like Solid.js
      • Discrepancies between commanded format and the current state
        • E.g. After changing code style: "JavaScript files should only export named entities, do not use default exports"
        • Possible solution: Do not command style, leave the code as generated.
    • Failure modes
      • Incorrectly specified prompt

        • "Render" -> set styles needed

          Without "Do not create new files" it tries to factor the common part out, but fails (its hard to factor out styles, especially without more context) |75%

      • Cognitive load, e.g. doing multiple things (e.g.: feature + refactor) in the same prompt
        • But often it works, e.g.:
        • Guess: Too much cognitive load leads to failing to give attention to part of the input
      • Solution: Clearly defined baby steps
  • Approaches to LLM Coding
    • One Prompt Generators (Automated Waterfall)
    • Iterative Systems
      • A few design questions
        • How to pinpoint relevant parts of large codebases
          • ctags
          • Embedding (Indexing) the source code
          • Manually
          • Map (refactor)
            • sweep ai
        • Interaction style (UX)
          • Focus on code (typically vscode based)
            • Rift
          • Focus on requirements (typically github based)
            • Sweep
          • Focus on Human-AI interaction (own UI or CLI)
            • aider (CLI)
            • Junior
        • How to construct prompts
          • From source code directly
          • Treat them as configuration
        • When and how to use error auto-feedback, prompt chains, actor/critic etc.
    • Junior Demo
      • Colin Raffel
  • Krisztián Schäffer
  • aijunior.dev
Loading page ...

Print options

Expand or collapse list branches

Want checkboxes? Change the list style

List style:

Display or hide list attributes

x

Expand & collapse ec

Import im

Examples and details

Word count wc

Current selection
Words: #{js-wc-sel}
Characters with spaces: #{js-cc-space-sel}
Characters without spaces: #{js-cc-sel}
The whole list
Words: #{js-wc}
Characters with spaces: #{js-cc-space}
Characters without spaces: #{js-cc}

List view options oo

  • hc
  • hf
  • sd
  • List style
  • om

Any email, forwarded to this address, will appear in beginning of this list.

Send an email to yourself and add the sender to Contacts for future use.

  • The email subject becomes the list item's text.
  • The email body becomes the list item's note.
  • All attachments from the email are attached to the list item (PRO only).
  • In the subject, you can also add #tags, ^due dates, and @assignees with Checkvist's smart syntax.

You can also set up voice integration on mobile devices