# Language Models in Programming #

LLMs today: How good are they?

- Solving coding challenges
   - 
![](https://checkvist-prod-uploads.s3.eu-west-1.amazonaws.com/u/1fkL4PbMyvIdMI/eaf60ac5e4c1b7076629c7d8f7168dab3e250779/preview/Screenshot_2023-09-18_at_9.54.59.png?X-Amz-Expires=3600&X-Amz-Date=20260705T164945Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPQDSU45IS7DDLBA%2F20260705%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=59e84b27f093aa2e7f35220c4b89526e81051fb5711ad70a303e77cd9480eccb)
   - Single prompt at its max:
![](https://checkvist-prod-uploads.s3.eu-west-1.amazonaws.com/u/0YvvmL35PcK0t7/fc64c0294377f603648683711d91c3ee2088eced/preview/Screenshot_2023-09-18_at_9.59.29.png?X-Amz-Expires=3600&X-Amz-Date=20260705T164945Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPQDSU45IS7DDLBA%2F20260705%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=34a85b4e4e7b32c6d1b10c94f0da0bc19a2f7903d2de8112ad9e163b01055558)
   - Source:
   - 
![|100%](https://checkvist-prod-uploads.s3.eu-west-1.amazonaws.com/u/4DXFzD465cy3Qq/d63567886af7774347a015e54b6fec3277f3943f/preview/Screenshot_2023-09-18_at_10.11.26.png?X-Amz-Expires=3600&X-Amz-Date=20260705T164945Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPQDSU45IS7DDLBA%2F20260705%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=2df7d655fb337fb43fc91222aadd8f8e02f8450aa49b343b93cad1b0aa41e73b)
      - https://arxiv.org/abs/2303.12712
   - New Bard vs GPT-4 (2023-09-20):
      - Using web technologies, create a simple 2D physical simulation of a solar system with one planet. The planet should orbit around the center of the system. 
- Set up a simple 20 fps game engine using setInterval()!
- Set up an area (a <div>) on the page for displaying the simulation! The planet will be displayed using an another <div> inside the area. You can avoid displaying the star.
- The model of the planet is its P = (x,y) position and V = (vx, vy) speed vector. The center star is fixed at C = (starx, stary) For the sake of simplicity, you do not need to use SI units. Use pixel coordinates directly!
- One step of the simulation is defined by the following formulas:
x +=  vx
y +=  vy  // The position is modified with the speed.
distance =sqrt((starx - x)2 +(stary - y)2)
force = G / distance2   // G is the gravitational constant
vx += force * (starx -x) / distance
vy += force * (stary -y) / distance  //The speed is modified with the gravitational force
- Set up the P, V and C vectors and G constant at the start of the simulation so that it looks nice. (E.g.:  C=(100, 100), P=(200, 100), V=(0, 1), G= 500)
      - GPT-4 solves it at first trial
      - Bard almost solves it, but the code throws errors (2x trial, different errors). After fixing by a new prompt, it works.
- Working memory (input token count, e.g. 4K, 32K)
   - In the kilobyte range (vs humans?)
   - They need some 'attention' mechanism to work on large codebases
- Cognitive load they can bear (attention head count (128+)  and dimension)
   - Number of distinct commands
      - Junior: 13 fixed  + task
   - How far the model must go from its training knowledge
      - E.g. coding in less popular frameworks like Solid.js
   - Discrepancies between commanded format and the current state
      - E.g. After changing code style: "JavaScript files should only export named entities, do not use default exports"
      - Possible solution: Do not command style, leave the code as generated.
- Failure modes
   - Incorrectly specified prompt
      - "Render" -> set styles needed

Without "Do not create new files" it tries to factor the common part out, but fails (its hard to factor out styles, especially without more context)
![|75%](https://checkvist-prod-uploads.s3.eu-west-1.amazonaws.com/u/IvsIXxbvSyr1n5/908dd5fabcf43382744ce5c50e43afeaeeeec275/preview/Screenshot_2023-09-16_at_21.30.43.png?X-Amz-Expires=3600&X-Amz-Date=20260705T164945Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPQDSU45IS7DDLBA%2F20260705%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=39feb65de4e1213943535ee8428bd9136ca7a4fddffe4fea644b17bbce89a58f)
   - Cognitive load, e.g. doing multiple things (e.g.: feature + refactor) in the same prompt
      - But often it works, e.g.:
![](https://checkvist-prod-uploads.s3.eu-west-1.amazonaws.com/u/YL3GSD4Z7BYOdJ/992cd908ef69ca01adf71905fb7121eb066cf842/preview/Screenshot_2023-09-18_at_16.11.08.png?X-Amz-Expires=3600&X-Amz-Date=20260705T164945Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIPQDSU45IS7DDLBA%2F20260705%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7103ef46128f45f25cf215bf2d84ba59d51d8bd982b8324109c1e19711f46aa2)
      - Guess: Too much cognitive load leads to failing to give attention to part of the input
   - Solution: Clearly defined baby steps

Approaches to LLM Coding

- One Prompt Generators (Automated Waterfall)
   - GPT-Engineer
      - https://github.com/AntonOsika/gpt-engineer
- Iterative Systems
   - A few design questions
      - How to pinpoint relevant parts of large codebases
         - ctags
         - Embedding (Indexing) the source code
         - Manually
         - Map (refactor)
            - sweep ai
      - Interaction style (UX)
         - Focus on code (typically vscode based)
            - Rift
         - Focus on requirements (typically github based)
            - Sweep
         - Focus on Human-AI interaction (own UI or CLI)
            - aider (CLI)
            - Junior
      - How to construct prompts
         - From source code directly
         - Treat them as configuration
      - When and how to use error auto-feedback, prompt chains, actor/critic etc.
- Junior Demo
   - Colin Raffel

Krisztián Schäffer

aijunior.dev