Introduction to Multimodal Reasoning

<p>We are building a visual diagnosis agent that consumes screenshots and user descriptions to reason about UI bugs and system errors. It combines vision and language understanding to generate structured incident reports without sending your data to closed-source platforms. Frontend teams and SREs can use it to triage issues faster by automating the first pass of root-cause analysis.</p> <h2 id="what-youll-need">What you'll need</h2> <ul> <li>Python 3.10 or newer.</li> <li>An Oxlo.ai API key from <a href="https://portal.oxlo.ai" rel="noopener noreferrer">https://portal.oxlo.ai</a>.</li> <li>The OpenAI SDK: <code>pip install openai</code>.</li> <li>A sample image file named <code>error_screenshot.png</code> in your working directory.</li> </ul> <h2 id="step-1">Step 1: Instantiate the Oxlo.ai client</h2> <p>I keep the client in a dedicated module so I do not repeat boilerplate. The base URL points to Oxlo.ai, and I use <code>kimi-k2.6</code> because it handles both vision and long-context reasoning in a single request.</p> <pre><code>from openai import OpenAI import os client = OpenAI( base_url="https://api.oxlo.ai/v1", api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY") )</code></pre> <h2 id="step-2">Step 2: Encode the image</h2> <p>Vision models on Oxlo.ai accept base64-encoded PNG and JPEG data inline. This helper reads a local file and returns the data URI string.</p> <pre><code>import base64 def encode_image(image_path): with open(image_path, "rb") as f: return base64.b64encode(f.read()).decode("utf-8")</code></pre> <h2 id="step-3">Step 3: Define the reasoning system prompt</h2> <p>I want the model to separate observation from inference. The prompt forces it to reason aloud before concluding and to output strict JSON.</p> <pre><code>SYSTEM_PROMPT = """You are a senior site-reliability engineer with expertise in frontend systems. When given a screenshot and a user report, reason step by step: 1. Observe what is visibly wrong in the image (errors, blank states, layout issues, console messages). 2. Correlate those observations with the user's text description. 3. Hypothesize the most likely root cause. 4. Propose an immediate fix and a prevention step. Respond in valid JSON with keys: observation, correlation, root_cause, immediate_fix, prevention."""</code></pre> <h2 id="step-4">Step 4: Build the multimodal request</h2> <p>The user message is an array of content blocks. One block carries the base64 image, the other carries the text query. Because Oxlo.ai uses request-based pricing, sending a large screenshot and a verbose system prompt does not inflate cost the way token-based inference does. You can compare plans at <a href="https://oxlo.ai/pricing" rel="noopener noreferrer">https://oxlo.ai/pricing</a>.</p> <pre><code>def diagnose_issue(image_path, user_description): b64_image = encode_image(image_path) response = client.chat.completions.create( model="kimi-k2.6", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{b64_image}" } }, { "type": "text", "text": f"User report: {user_description}" } ] } ], response_format={"type": "json_object"} ) return response.choices[0].message.content</code></pre> <h2 id="step-5">Step 5: Parse and display the result</h2> <p>The raw response is a JSON string. I load it and print each reasoning stage so the output is readable in a terminal.</p> <pre><code>import json def run_diagnosis(image_path, user_description): raw = diagnose_issue(image_path, user_description) result = json.