AbstractPhila PRO

The space is being updated to reflect the new experiment successes. Caption and tags are assistants because Lucia is a smart and capable girl, additionally with Lucia most of the others can also reflect the behavior, or their base forms used to caption the image prior to running the COT image captioning system.

Even qwen 3.5 0.8b is capable of a miniature caption for 9b to process into something thought-related, so these models simply needed a couple tweaks, they work with the correct provocation.

The huggingface space is available to play with for the showcase of potentials of each of the models. Some like Qwen 3.5 9b are able to read text rather well, others not so well. Be wary as you experiment.

https://huggingface.co/spaces/AbstractPhil/think-symbolic-images-testbed

They are powerful models already. Now they produce symbolic thinking captions.

MORE models are being added as Claude prepares them for the space. My apologies If the space restarts during your testing, the process is ongoing.

The current line of models have weaknesses with human subjects currently. Not to worry though, progress my friends. Progress.

I would NOT train models with human subject data from this model yet. The models require refining and reinforcement preparation for correct and incorrect objective responses. That's fine though, there's no shortage of data to reinforce and test.

I'm going to try some tweaks. Likely if I feed the model some hints, like say from a booru tagger, the model may be more apt to respond correctly.

Okay, so all she really needs is a hint at the prompt in general.

If you feed the image's original prompt in like so;

Look at this picture and think it through in writing, a closer look; then write a caption and a prompt or tags, and the questions a careful looker would still ask, each as `question: what -> answer [record, chain or unknown]`.

You get bland COT outputs. However, if you do this;

Look at this picture and think it through in writing, a closer look; then write a caption and a prompt or tags, and the questions a careful looker would still ask, each as `question: what -> answer [record, chain or unknown]`.
Helper Prompt:small image prompt goes here
Helper Tags:any booru tags go here

This tends to solidify the system much more effectively, as the model herself is quite large and quite intelligent thanks to the base model. She doesn't need much, just a nudge. The COT results from this have been considerably more effective and diverse.

It's not a guaranteed fix until I alter the training process a bit and retrain her, but for now the tweak will work to activate the Qwen 9b VLM inference a bit more effectively.

We're folding Gemma 4's VLM system into the mix. The net can be cast widely and the data can be diversified via many sources rather than just a handful.

The Qwen 3.5 symbolic image thinkers will be known as Lucia.

Haven't picked a name for Gemma yet. We'll get started and see how well the model performs by the get-go, full statistics and target information accumulation like the qwen line.

Experimental flagship; Qwen 3.5 9b Lucia
Experimental faster; Qwen 3.5 4b tobenamed
Experimental fastest; Qwen 3.5 0.8b tobenamed -> the non-thinker

Almost all major experiments performed using this model has been successful or semi-successful thus far. The original writers for the data are providing enough inference and information for the model to begin THINKING in the image words when seeing pictures, not just guessing or relating to miniature pieces of that image. THINKING about the STRUCTURE of that image, the PURPOSE, the MEANING BEHIND that image.

Lucia, the one who thinks in images.

Our rotary system /NEEDS/ a teacher. A mentor that grounds the model in reality so the collective can learn effectively.

If we ABSOLUTELY require it, we can utilize Qwen 3.8 27b, but that will run considerably slower and train considerably slower. So 3.5 9b is the target for now. If a better model between now and then is discovered or emerges, we will utilize that model.

Reasons:

1. Qwen 3.5 9b is effective enough as an individual model to finetune as necessary into any shape the collective will require.
2. Takes up about 10 gigs of vram, so the model is compact enough to fit on most cards.
3. Qwen is used commonly for many projects; Qwen Image Edit 2.1 being a particular choice, ideal for testing the direct capacity of Beatrix.
4. Qwen is stable and intelligent.

I had a great deal of fun when playing with Qwen Edit 2.1 using the primary, turbo, and playing with the lightning lora as well. The lightning lora seems to snip out some pixels on the left for editing so I'll need to figure out what's going on with that or if it just does it by default.

Our release finetune will be designed to think in images. Very different than a prompt synthesizer. Trained full of key phrases and anchor information, specifically trained to think in images to produce the exact format of information we require to frame for our upcoming media curation collective.

A master who thinks in media is required, and with that we need the intelligence multiple Qwen 3.5 9b finetunes will bring to the table to train our pieces.

Finetunes to extract data for pretrain data that doesn't exist, basically.

The T2I and I2I of Qwen vision's prompt rewrite versions of the 9b were both tested, neither of them produce the necessary shape our data requires yet. However, we're getting closer day by day.

Qwen 3.5 9b's VLM should be enough to generate a full dataset system for the rows. Sonnet is going to take weeks at this rate, so I'm testing some solidifications.

A text only lora KIND OF worked, but it wasn't strong enough. Also ~900 image rows is clearly not enough. I'm going to need a full synthetic symbolic engine if I want something of legitimate reuse.

It SHOULD be powerful enough to workhorse a full pipeline structure when it's ready. Qwen 3.5 9b is pretty powerful.

The model training both PEFT lora or full finetune fit snugly on a colab RTX 6000 pro, so it'll train pretty quick. I ran rotary for 8 days on them, this little prototype should be done in a day or two with any luck. Then a large scale extraction, then we can start teaching some experts some legitimate capacity.

When prepared it goes in the list of prepared utilities released by the AbstractPowered systems. I've been putting off compounding them, but I plan to finish the compounded list and release the handful of valid utilities to the AbstractPowered as release models, and likely next week or the week after I'll prep the AbstractPowered research line sector.

I'm going to have the claudes continue generating the image chain of thought system while I sleep.

There are still more steps between here and training the next model, one of which involves image thought being in the experiment line. This will include a form of VIT, a form of diffusion, and a form of traditional classification experts.

I'm thinking around 20,000 should be enough to train a fair-sized qwen for the task. It's only about 4,000 so far, so it'll take a few more weeks of off and on. I'll try a finetune here or there to see if it gets better, then run a couple datasets to compare the results

More than likely the better qwen will need a task trained VLM lora, which I've never built a combination vision + text lora so that'll be interesting. There's probably a series of losses that make sense more than just using the clip cross-contrast so I'll need to research a bit.

Extraction with one of the qwen 8b or 9b models should provide a good enough result to train another model, but it's still just going to be qwen results tuned to the shape unless I run a few other processes. Qwen 3.8 should be strong enough, hopefully, but I'm not even sure if 3.8 has a VLM yet so I'll have all that worked out sooner than later.

We have our rotary selector for the next version of the model, Dante.

The current concept that I've dubbed multi-rotary pairings, extends to a solid context window space of around 14 million or so pairs so far. This is conjunctive using different sized pairings between 2 and 16, as per the aleph set mathematics capacity allows.

There is a bit of a rift that forms still at this scale, but the rotary itself can easily encompass framed sets of 14 million bytes worth of paired frames, treated as though they are individual lookups thanks to the prototype "AlephFrame" rotary system.

This doesn't merit a full conjunctive LEARNING capacity, but it does converge a structural guarantee capacity.

This frame is a little different than standard attention, but it'll snap right to models. So far the pairings are a little wild on some edge cases, but I have another upcoming test that I believe will resolve that hyperbolic rifting.

Currently we're at around a 0.00001% invalid addressing fault, which when scaled up is a bit substantial. However, with additional pairings the system cleans up the bpb and the output cleans.

Essentially this means entire context window can be a lookup table, so that's pretty quick at least. The models with their arms train at around 750k byte/s for now, and the scaled up MOE variant will train about as quick as you wish to train the experts for.

Essentially, what we're looking at here, is a cascade solidity system with independent experts that can be removed and replaced, simply existing within that space defined. Within that spatial geometry, it is completely cut off from any external stimuli, and that space is perfectly preserved from the external state of the model.

Meaning, you can simply snap something out and put a new interface in. Small interface, big interface, doesn't matter. Input, output.

We'll be targeting the space over 1 million context window, which should help solidify the system's internal shared structural space.

Very strong signals for full decoupled fixed position rotary. Huge amounts of experimentation done.

Lets figure out how 1 million context window works, shall we?

https://huggingface.co/AbstractPhil/geolip-rotary-experiments

I know this likely sounds odd or a bit too generic currently. The concept here is straightforward in pragma though. Fixed position, dynamic position, rotary position, global, local, or any other doesn't matter. The model simply needs that learned behavior and the understanding of how to seek the information, with a stable protocol system for acceptance, rejection, and so on. You don't need 600 layers in a model to debate the answer to this question internally, trained through 1000 step differentiated Adam, when the stable answer is something represented somehow in something.

In our case we take a piece of the context window from the overarching global system, say a dynamic allocation <FRAME>, and we snap a positional framed request to this. One of the collective with the necessary compositional addressing behavior intercepts this request, processes the information within it's bounds benefitting from the core model as necessary, sends the compositional result back.

One token done. Say we do this 500 times, each compositional structure intercepted by the necessary arm's behavior, providing that learning as a standard MOE would, except with a format of purpose.

Each of those compositional leaders have their own sub-collectives of solvers. Whatever they distilled or utilized or whatever the model is constructed to use, something like RNN, memory, or whatever. The next token or tokens are predicted and fed out utilizing the necessary structure within those dynamic framings, and once the capacity for that frame is exhausted the frame rises.

Simple in texture really. Something akin to a programming stack, except the addressing for the stack's behavior is a complex mathematic rotary combinatory behavior. We've been doing this for 80 years.

Unknowns are seen and yielded. Problems are spotted and yielded. Everything between is fielded, tested, and internally debated.

This when functional, will not need a byte model. This structure will behave effectively with tokenization models as well. The lookup system needs maturing but the system will comply.

This may seem like 5 steps back 1 step forward from just feeding a large model all the necessary information and getting a multi-debated answer over time. A seemingly difficult and complex idea to understand forming a lesser structure.

This isn't exactly the case though. In this model design's case, you gain the power of yielded multi-stage, multi-step, pre-planned arbitration, in a stable next-token prediction system.

Our hardware is TERRIBLE at this sort of thing, so I have a few pieces to make.

You don't WANT a model like tiny Beatrix to use a system like this, you need something with a bit of intelligence to arbitrate the subsystem. A Qwen with some oomph is a good option.

So... yeah the little models seem to learn most text related tasks on this scale. Individual structures with training runs seem to have very little problems learning them.

9 million params isn't very much, but it's more than enough for this task apparently. I'll be finetuning something much stronger with the dataset sonnet, opus, and fable engineered for the image thinking task. I'm thinking a powerful qwen or something like that. Something that knows images already.

That should be an appropriate target. Something with more brains than an atom.

Castor learned clean chain of thought.

Got a bit overzealous on my fable, opus, and sonnet usage - it's pretty much gone. I'll be setting up something with Opus but the usage is limited so nows a good a time as any to take a break until tomorrow.

Pollux should finish soon, then the dual thinking nanos should be functional.

Pollux finished.

Experimental thought dataset should be pretty beefy now. Not sure how well it'll work unless I train a vit with the information using cross-contrast, so I'll run some finetunes and benchmarks. The nano models don't understand thinking, so I'm not really sure what'll happen if I just raw feed it. I assume overfitting from just the data if I run multi-epoch due to the lack of CoT training, so the model's CoT will likely need to be conditioned ahead of time to be generic CoT as per my assumption. This isn't a guaranteed requirement, it may very well be the opposite - where CoT can be taught using symbolic objective representation rather than needing some sort of CoT dataset to baby-step into the system.

I'll run some experiments and tests, then figure out some good methodologies for the trie structure.

nano-pollux-dual-v1.2 is done, full frame train is in.

Model eats them up. Control is the nano-castor model having never seen frames.