🌌 Illustrious Studio Engine

🚀 Overview

This repository contains the dynamic, fault-tolerant Google Cloud orchestration matrix for the Illustrious Studio Engine's spatial rendering workers. It completely automates the lifecycle of headless, GPU-accelerated (Nvidia Tesla T4) Spot instances running the Lumina2/Juggernaut-Z diffusion pipeline.

Instead of burning cash on idle compute, this system implements Zero-to-One Auto-Scaling. It tracks hardware availability, dynamically hops across geographic zones to find cheap Spot capacity, seeds persistent model storage on-the-fly, and securely proxies frontend HTTPs traffic down to raw backend worker sockets.

---

GOALS

1. There is this map set https://lvlworld.com/review/id:1937 that has both a winter and summer version. So i need to make a list of every difference, like one of those eye-for-detail games where you compare two images. Once i have that list, i will build the studio that does each of those things. Using generative neural networks in a cloud-gpu and web workers for the smaller tasks like code edits and model insertions using an LLM in a background worker and customized LORAs. this process gives me a solid scoped path for what tooling to build
2. Import models dynamically from https://github.com/allenai/objaverse-xl lay them out using a custome spatial LORA that does things like insert a bicyle behind the bench.
3. Convert quake 3 maps to point clouds, and add horror elements and flashing lights automatically. like the old levels are breaking down from not being maintained and its possible to fall from one world into another world and eventually into the depths of quake 3 hell.
4. Add BSP, Play Canvas voxel pipeline, and depth based overdraw culling to this example https://sparkjs.dev/examples/splat-painter/index.html
5. one query to the LLM should completely change a world. need to find a technology for pattern matching and texture generation so ceilings become a texture for use in the vector based (as opposed to gaussian based) world.
6. Target - speaking in to a voice LLM and tell minecraft to build any object in objaverse.
7. TODO: enabled cache reserve, start an HF account, or switch to google for distribution, whichever is cheaper. Can't have 1,000 people downloading 300MB out of my GitHub account probably.

So Far

07/01/26

Fair call. Packing everything into one generic global mega-atlas texture is what broke your asset explorer and submodels.

Looking at Screenshot 2026-07-01 224455.png, the Asset Explorer previews are broken because every single individual material is sharing that exact same mega-atlas uniform. Since the material preview geometry doesn't have the custom scaled atlas vertex attributes, it samples the whole map's shadow layout scaled down across a tiny sphere.

Looking at Screenshot 2026-07-01 224400.jpg, your map's main brush surfaces render right, but external submodels (like the blacked-out statue or sub-bsp assets) are completely ruined because they are referencing fallback or unmapped lightmap texture handles.

To do this right and restore structural stability, we should drop the mega-atlas complexity and re-integrate individual, granular lightmap textures exactly like Toji's codebase did.

---

1. The Worker Code: Parse Individual Lightmaps

Instead of packing pixels into a massive master sheet, parseRawBSP should keep lightmaps separated as an array of individual 128x128 RGBA buffers.

Update that section in parseRawBSP to match this layout:

let lmLump = lumps[14];
let lightmapSize = 128 * 128;
let lmCount = lmLump.length / (lightmapSize * 3);

let bspLightmaps = []; // Flat array holding separate individual 128x128 buffers
let lmPtr = lmLump.offset;

for (let i = 0; i < lmCount; i++) {
    let elements = new Uint8Array(lightmapSize * 4);

    for (let j = 0; j < lightmapSize * 4; j += 4) {
        let r = view.getUint8(lmPtr++);
        let g = view.getUint8(lmPtr++);
        let b = view.getUint8(lmPtr++);

        // Quake 3 brightness adjust factor (4.0)
        r = Math.min(255, Math.floor(r * 4.0));
        g = Math.min(255, Math.floor(g * 4.0));
        b = Math.min(255, Math.floor(b * 4.0));

        elements[j]     = r;
        elements[j + 1] = g;
        elements[j + 2] = b;
        elements[j + 3] = 255;
    }

    bspLightmaps.push(elements);
}

Since we are using individual lightmap textures, stop multiplying the coordinates by atlas scales. Inside the face processing loop where you compute coordinates, assign the raw lmCoord coordinates directly out to the vertex:

// Inside your face loop processing for face.type 1, 2, and 3:
vert.lmNewCoord[0] = vert.lmCoord[0];
vert.lmNewCoord[1] = vert.lmCoord[1];

And make sure your face index mapping tracking maps the target index over to the final extracted surface object inside extractSurfaceGeometry:

// Inside extractSurfaceGeometry, make sure to forward the face's target lightmap index:
return {
    index: s,
    shaderName: surface.shaderName,
    geomType: surface.geomType,
    lightmapIndex: surface.faces && surface.faces[0] ? surface.faces[0].lightmap : 0, // Track original index
    positions: new Float32Array(subPositions),
    normals: new Float32Array(subNormals),
    uvs: new Float32Array(subUvs),
    lightCoords: new Float32Array(subLightmapCoords), // Extracted un-scaled coords
    colors: new Float32Array(subColors),
    indices: subIndices
};

---

2. Main Thread: Hydrate Individual Texture Instances

When storing the lightmaps data array on your loader instance, map the raw byte buffers into individual, decoupled THREE.DataTexture array instances:

setLightmaps(rawLightmapsArray) {
    this.lightmapTextures = [];

    if (Array.isArray(rawLightmapsArray)) {
        rawLightmapsArray.forEach((bytes, index) => {
            const texture = new THREE.DataTexture(bytes, 128, 128, THREE.RGBAFormat);
            texture.wrapS = THREE.RepeatWrapping;
            texture.wrapT = THREE.RepeatWrapping;
            texture.minFilter = THREE.LinearFilter;
            texture.magFilter = THREE.LinearFilter;
            texture.flipY = false;
            texture.needsUpdate = true;

            this.lightmapTextures[index] = texture;
        });
    }
    return this;
}

---

3. Associate and Bind Prior to Generation

Now, your individual meshes can request their exact localized shadow layer block inside processBatch, and your Asset Explorer can gracefully safely ignore texture configurations for unrelated contexts.

Update processBatch to pick out the correct specific index:

processBatch(surf, rootNode) {
    // ... (Your standard geometry buffer configuration block goes here)

    // Select the unique lightmap texture assigned specifically to this geometry piece
    let faceLightmap = this.whiteTexture;
    if (this.lightmapTextures && this.lightmapTextures[surf.lightmapIndex]) {
        faceLightmap = this.lightmapTextures[surf.lightmapIndex];
    }

    // Set it cleanly on the builder before compiling the active stages
    if (typeof this.materialBuilder.setLightmap === 'function') {
        this.materialBuilder.setLightmap(faceLightmap);
    }

    let textureShaderPath = surf.shaderName;
    let finalMaterials = [];

    // Check if it's a standard interface preview pass or an empty shader name
    const isAssetExplorer = textureShaderPath.toLowerCase().includes('preview') || textureShaderPath === '';

    if (isAssetExplorer) {
        // Force asset previews to fall back cleanly to a solid white lighting multiplier
        if (typeof this.materialBuilder.setLightmap === 'function') {
            this.materialBuilder.setLightmap(this.whiteTexture);
        }
    }

    if(textureShaderPath && textureShaderPath !== "noshader") {
        let lookupName = textureShaderPath.toLowerCase();
        let cachedShader = registry[lookupName];

        if(cachedShader) {
            finalMaterials = this.materialBuilder.buildMaterials(cachedShader);
        }
    }

    if(finalMaterials.length === 0) {
        finalMaterials.push(this.materialBuilder.buildDefaultMaterial(textureShaderPath, THREE.DoubleSide));
        finalMaterials[0].uniforms.lightmap.value = faceLightmap;

        this.materialBuilder.resolveTexture(textureShaderPath + ".jpg", false, (tex) => {
            finalMaterials[0].uniforms.map.value = tex;
        });
    }

    finalMaterials.forEach((material, stageIndex) => {
        let surfaceMesh = new THREE.Mesh(geometry, material);
        // ... (remaining mesh generation attachment logic)
        rootNode.add(surfaceMesh);
    });
}

This restores the isolated, pure single-pass texturing system from Toji's codebase while cleanly fixing the asset inspector previews.

06/29/26

Need to figure out why empty boxes are coming up from "house" search instead of the models. Need to test freshly trained LoRa model because i added scaling and volumetric rotation.

searchWorker.postMessage({
	type: 'SEARCH_QUERY',
	baseURI: window.location.origin + '/',
	parquetFiles: [DEFAULT_PARQUET],
	payload: "house"
});

and

window.Editor.addVisualModelToNunuAssets(conversionResults[4]);

06/17/26

Got ONNX and wLlama models loaded inside a web worker initiator, even though wLlama comes with multi-threaded worker support out of the box. Both of them are loading multiple em-pthreads as workers from inside the initial worker, that reports back to the front-end UI.

I am starting to formulate training data to apply a lora and grammar combination to a gguf that can extract specific elements from language that have specifically to do with orientation related to a base [0,0,0] or related to another object in the description, here's what i forumaled with the help of my highly trained AI

---

Spatial Lora Training

0. The Spatial Breakdown:

[  X,  Y,  Z,  Pitch,  Yaw,  Roll,  Scale  ]
└─ Position ┘   └─── Angle ───┘   └ Size ┘

• X (hw): Moves it forward/backward.

• Y (-hd): Moves it left/right.

• Z (hh): Moves it up/down.

• Pitch (0): Nods/tilts forward or backward.

• Yaw (0): Swivels/twists left or right.

• Roll (0): Leans or banks side-to-side.

• Scale (0.5): Multiplies the overall object volume uniformly. In this case, shrinking the object down to half its normal size. (Note: If this field is a nested array like [1.2, 1.2, 0.8], it scales the Width, Depth, and Height independently).

Two stone pillars holding up a heavy steel crossbeam mesh across the top.

Output: [cylinder][-3fw,0,0,0,0,0,1.5][cylinder][3fw,0,0,0,0,0,1.5][mesh][@0,@1][0,0,fh,0,0,0,[6.2,0.4,0.2]]

1. Dimension & Anchor Aliases (The Bounding Variables)

This comprehensive list maps natural language spatial concepts directly to your token-minimum layout blocks, accounting for your new distance multipliers (4fw, 4fh), relative directions, and compound placement rules. These represent the dynamic dimensions, global overrides, syntax hooks, and primitive keys utilized by the layout engine.

parseSpatialCommands('[red balloon][0,0,0,0,0,0,1][elephant][0,fd*@idx,0,0,0,0,1]')

Its an implied 1fw*@idx would be adding 1 full width of the previous, as opposed to -/+@idx or @idx-fw precise translations would be the same as subtracting a full width just like the scalar, but it also allows for exact values like @idx-100 would be the previous object x - 100 units

---

2. Explicit Multiplier Conversions (The "Near & Far" Rule)

To make translation absolute, anytime the user specifies human-scale distance descriptors, the model maps directly to exact bounding multipliers rather than guessing arbitrary float boundaries.

A. Distance Multiplier Mapping Table

B. Structural & Boolean Interaction Mapping Table

C. Complex Scene and Absolute Positioning Table

D. Primitive Shape Target Table

E. Structural Deformation Mapping Table

---

3. Examples

A. Geometric Primitive Vocabulary Mapping

B. Spatial Direction & Distance Syntax Mapping

Each block structure follows your exact vector format: [X, Y, Z, Pitch, Yaw, Roll, ScaleFactor].

---

Vertical Layouts (Z-Axis)

Lateral / Proximity Layouts (X-Axis)

Depth & Occlusion Layouts (Y-Axis)

Diagonal & Compound Layouts (Multi-Axis)

---

C. Orientation & Scale Descriptors (Rotational Syntax)

These handle terms indicating posture, direction faced, or sizing variations relative to the anchor block.

---

Example A: 3-Model Relational Chaining (The Stack Pass)

• Input: “A table, a plate on top of the table, and a cup next to the plate.”
• Deconstruction: Object 1 (plate) anchors to Object 0 (table). Object 2 (cup) must anchor specifically to Object 1 (plate), not the table.

Example B: Non-Relational / Absolute Positioning Override

• Input: “An elephant at the center, a mouse way off to the right 10 units away, and a bird hovering 5 units directly above the elephant.”
• Deconstruction: The mouse ignores the elephant's footprint and jumps out by fixed units. The bird passes a pointer back to the elephant (@0) to calculate its height position, skipping the mouse entirely.

Example C: The Midpoint Interpolation (Between Rule)

• Input: “A couch on the left, a chair on the right, and a small rug placed right between them.”
• Deconstruction: The parser computes the absolute distance between Object 0 and Object 1, splitting the vector difference down the exact midpoint line.

---

4. Response Fitting Grammar: Gerganov Backus-Naur Form

# Root: one or more spatial blocks, optional trailing whitespace
root ::= block_sequence ws?

block_sequence ::= block (ws_or_nl block)*

block ::= primitive anchor? vector

primitive ::= "[" [a-zA-Z0-9_-]+ "]"

anchor ::= "[abs]" | "[@" [0-9]+ ("," ws? "@" [0-9]+)* "]"

# Vector now supports 3-7 elements with nested scaling and expressions
vector ::= "[" value ("," ws? value)* "]"

value ::= expression | scaling | attribute | num

# Improved expressions: handle negatives, multiplication, @idx, sym(), etc.
expression ::= sign? term (op term)*
term ::= variable ("*" (variable | num))? | "sym(" ws? expression ws? ")" | num
variable ::= "fw" | "hw" | "fd" | "hd" | "fh" | "hh" | "@idx"

scaling ::= "[" sign? num "," ws? sign? num "," ws? sign? num "]"

attribute ::= "noise=" num | "mesh"

num ::= sign? [0-9]+ ("." [0-9]+)?

op ::= "+" | "-" | "*" | "/"

sign ::= ("+" | "-")+

ws ::= [ \t]+
ws_or_nl ::= [ \t\r\n]+

---

5. Lora Training: Message Format

A. Combinitorial Prompt Training Messages

• sme-gemini-combos.json

B. Inset Complex Training Messages

• sme-gemini-complex.json
• sme-gemini-multiples.json

C. Positonal Training Lora Messages

• sme-gemini-positional.json

D. Negative Feedback Training

• sme-gemini-negatives.json

E. Objaverse Noun Extraction Training

• sme-gemini-nouns.json

F. Complex Queries and Room Building

• sme-gemini-rooms.json

G. Terrian and Outdoor Generations

• sme-gemini-scenes.json

H. Camera Coordination for Scene Changes

• sme-gemini-scenes.json

---

So Far (Continued)

06/15/26

That repository is an absolute goldmine for this specific use case. Leveraging Project Euler solutions written across dozens of different languages means you get highly representative, idiomatic structural code blocks solving the exact same algorithmic math problems.

By restructuring that repository into your conversational JSON format, you are building a highly effective instruction alignment set.

---

Expanding Your Neural Patch: ASI and Spell-Checking

Once you have this baseline classification working, your intuition about expanding into ASI (Automatic Semicolon Insertion) or Syntax/Spell-Checking is spot on. You can use the exact same base model (Qwen2.5-0.5B) and target the exact same low-rank attention paths (["q_proj", "v_proj", "k_proj", "o_proj"]) to execute those specific code transformations.

The only things that need to change are your System Instruction and the Target Completion Template.

1. The Automatic Semicolon Insertion (ASI) / Code Formatter Patch

Instead of asking for a single-token identifier name, you train the model to look for raw, minified, or loosely styled snippets and complete them with structural precision.

{
  "messages": [
    {"role": "system", "content": "Analyze the input code. Enforce strict line-terminating semicolons, fix indentation boundaries, and return only the corrected source text."},
    {"role": "user", "content": "const x = 5\nlet y = 10\nconsole.log(x + y)"},
    {"role": "assistant", "content": "const x = 5;\nlet y = 10;\nconsole.log(x + y);"}
  ]
}

2. The Code Spell-Check / Typo Correction Patch

You can pollute your training inputs with deliberate typos in variable declarations, common property names, or method invocations, and provide the exact clean compilation target.

{
  "messages": [
    {"role": "system", "content": "Locate typographic errors in keywords, syntax elements, or method identifiers. Output only the corrected code snippet."},
    {"role": "user", "content": "functon calculateTotal(prce, quantty) {\n    return prce * quantty;\n}"},
    {"role": "assistant", "content": "function calculateTotal(price, quantity) {\n    return price * quantity;\n}"}
  ]
}

---

The Big Architectural Advantage

Because your chat-agent.py server is now using a compiled, unified matrix fusion step on startup (peft_wrapper.merge_and_unload()), you can build an array of independent micro-adapters on your desktop drive:

• loras/code_classifier_lora
• loras/code_asi_formatter_lora
• loras/code_spellcheck_lora

Whenever you want to switch your agent from being a lightning-fast router to an inline code-linter, you just update your disk variable pointer path, restart your server instance, and let it seamlessly rebuild its base layer paths. You get tailored, enterprise-grade tooling adjustments inside a 500-million parameter model footprint that runs natively on standard computer hardware.

06/14/26

🏗️ Architecture & Core Components

The backend is decoupled into three distinct microservice routes (runnable locally for debug or deployable as separate Google Cloud Functions):

1. clusterManager (The Brain)

• Endpoint: /api/cluster/status
• Role: Tracks the live state of the compute horizon.
• Mechanics: * Queries GCP for running or staging worker instances (illustrious-juggernaut-worker-node).
  • If the pool is empty, it triggers the bootGpu service to scale up.
  • Failover Engine: If the default zone (us-central1-a) is exhausted of Spot GPUs, it automatically pivots the entire infrastructure pool to backup zones (e.g., us-central1-f, us-east1-c).
  • Diagnostic extraction: Intercepts serial port boot logs for crashed machines to pass raw kernel/preemption errors down to the client canvas.

2. bootGpu (The Muscle)

• Endpoint: /api/cluster/allocate (Usually triggered internally by clusterManager)
• Role: Provisions the physical hardware, handles disk orchestration, and bootstraps the ML environment.
• Mechanics:
  • Image Recon: Dynamically queries Google's public registries to find the latest valid debian-11 OS image, bypassing hardcoded deprecation traps.
  • Mode A (Seeding): If the permanent Juggernaut-Z vault (illustrious-juggernaut-z-vault) is missing or unformatted in the target zone, it spins up a temporary seeder node, formats the SSD to ext4, pulls the 6B parameter weights directly from HuggingFace, and self-destructs.
  • Mode B (Production): Once the disk is READY, it attaches the vault in READ_ONLY mode (allowing concurrent multi-node attachment) and boots the GPU worker with an injected Python FastAPI agent.

3. spatialRelay (The Network Bridge)

• Endpoint: /api/spatial/relay
• Role: Bypasses browser Mixed-Content (HTTPS -> HTTP) security blocks.
• Mechanics: Acts as a secure, streaming pass-through. It accepts multipart/form-data (spatial coordinates and image slices) from the secure frontend canvas, pipes it directly into the raw IP of the active worker node, and streams the generated JPEG binary back to the user seamlessly.

4. worker-agent.py (The Artist)

• Role: The headless FastAPI inference engine running directly on the GPU node.
• Mechanics: * Loads RunDiffusion/Juggernaut-Z-Image into VRAM at fp16 precision.
  • Accepts spatial coordinate maps (X, Y, W, H) and base image slices.
  • Executes the diffusion pipeline and composites the generated "infected layer" back into the staging matrix before streaming it back.

---

⚔️ The War Room: Challenges Solved

We didn't just write a script; we fought through the GCP hypervisor to make it bulletproof. Here is the historical ledger of dragons slain:

• The SDK Client Migration: Bypassed the legacy new Compute() monolithic wrapper (which threw not a constructor errors locally) and successfully migrated to the modern Gapic sub-clients (InstancesClient, DisksClient, ImagesClient).
• The Auth Matrix Bypass: Solved Windows local Application Default Credentials (ADC) failures by extracting active OAuth2 session tokens from the Express middleware and piping them directly into the Google Cloud API constructors in real-time.
• The OS Image Deprecation Trap: Google quietly deleted the common-cu121-debian-11-py310 image families. Built a dynamic resolveActiveImage scanner that queries multiple projects to ensure the system always finds a bootable OS.
• The Ghost Disk Locks: Fixed a race condition where GCP threw a 400 Bad Request: resourceInUseByAnotherResource. The manager now actively scans for "zombie" seeder disks that failed to detach properly, issuing asynchronous purge commands to shred them before attempting a new worker boot.
• The ZONE_RESOURCE_POOL_EXHAUSTED Drought: When Google ran entirely out of cheap Tesla T4 cards in us-central1-a, the script threw cryptic serial port lockouts. Built a multi-region failover matrix (resolveOperationalZone) that actively queries hardware availability and shifts the infrastructure to a new zone instantly.

---

🔮 Future Plans & Next Steps

1. Frontend Canvas Hookup: Wire the React/Web application to consume the CLUSTER_MANAGER_URL and map the visual hardware staging states (STAGING, INITIALIZING_STORAGE, ACTIVE).
2. Telemetry Dashboards: Add a real-time log ingestion view to the front-end so users can watch the worker.log apt-get and pip installation streams as the node boots.
3. Scaling the Horizon: Currently hardcoded to GPU_COUNT: 1 and a single node setup. The READ_ONLY disk architecture supports dozens of simultaneous nodes—ready to be upgraded to a load-balanced array once user demand spikes.
4. Automated Reaper: Implement a background cron-job that pings the clusterManager to kill any workers that have been idle for more than 15 minutes to aggressively protect billing quotas.

---

Built for the Illustrious Studio Engine.