How IISc Bengaluru fine-
tuned vision models to read
hand-drawn sketches

Background

Closing the sketch gap in open-weight sketch vision models

The Opportunity

What the research design demanded from compute

Training a model that doesn’t yet exist is harder than scaling one that does. For O3SLM to work, three things had to come together at once, and each one stressed the underlying compute in a different way.

No training data existed. 

Every public sketch dataset was either too small or too narrow to train a model of this size on. The IISc team had to generate sketches automatically from photographs at scale. The pipeline ran continuously and produced 33 million sketch-image pairs before training could begin.

The model architecture was memory-hungry. 

O3SLM processes sketches, photographs, and language together in a single pass. Running this in training requires substantially more GPU memory than most shared GPU clusters can provide, and any compromise on batch size or model size would have produced a weaker model, particularly at the 13B parameter scale.

Reaching state-of-the-art needs many training runs. 

The team had to evaluate across four separate sketch datasets and repeat experiments across architectural variations to isolate what actually drove performance. Slow iteration would have meant a smaller, narrower paper.

To pursue all three at once, IISc needed dedicated, high-memory compute available on demand, without the queue contention that defines most shared GPU environments.

The Sketch Vision Model Solution

Two open-weight sketch vision models, built on Neysa Velocis

IISc ran every phase of O3SLM’s development on Neysa Velocis, from synthetic data generation through final evaluation.

The pipeline had three stages:

Synthetic data generation. The team built an automated pipeline that converted natural images into corresponding sketches, producing 33 million sketch-image pairs. The pipeline ran continuously in the background, generating the training corpus that no public dataset could provide.

Two-stage instruction tuning. Starting from LLaVA-v1.5 at both 7B and 13B parameters, O3SLM was fine-tuned on 815,000 multimodal instruction pairs in two sequential stages, allowing the model to learn both visual grounding (what is in a sketch) and instructional reasoning (what to do with it).

Evaluation across four benchmark datasets. The team validated O3SLM against the major sketch benchmarks in the field, running repeated experiments across design variations to confirm state-of-the-art performance at both model sizes.

The output is a pair of open-weight models, 7B and 13B, that treat sketches as a first-class input modality rather than as low-quality photographs.

The Neysa Partnership

Compute headroom that matched the research design

IISc’s deployment on Neysa Velocis was built around dedicated bare metal GPU infrastructure with the memory and throughput O3SLM’s architecture demanded. The Neysa team configured the environment to support all three stages of the project without forcing the research team to cut corners on model size, batch size, or experiment count.

Dedicated high-memory GPU nodes. Bare metal Velocis nodes gave IISc the GPU memory headroom to run O3SLM’s full architecture in training, with sketches, photographs, and language processed in a single pass. The team trained both the 7B and 13B versions at full size, with no compromises forced by hardware limits.

Sustained throughput for large-scale data generation. The 33-million-sketch pipeline ran continuously alongside training and evaluation workloads. Velocis carried the parallel demand without throttling either workload, keeping the data pipeline ahead of training.

Fast iteration for evaluation and experiment cycles. Instant access to dedicated compute let the team run repeated experiments across the four benchmark datasets and multiple architectural variations. The iteration speed compressed what would otherwise have been a multi-quarter evaluation cycle.

The Impact

Two open-weight sketch vision models, released for the field

World’s first open-weight sketch vision model. O3SLM is the first open-weight model that reads hand-drawn sketches as an input modality. The weights are released for the research community to build on, in both 7B and 13B versions.

Accepted at AAAI 2026. The work passed peer review at one of the most competitive venues in AI.

State-of-the-art results against open and closed-source models. Both the 7B and 13B versions outperformed every open-source rival and beat closed-source models like GPT-4o and Gemini 1.5 Pro across the four major sketch benchmarks in the field.