컴퓨트로늄 컴퓨트로늄

Why? Multi-tenant environments. First, we need to understand a few differences between environments:

• End-user UI
• Agent Runtime Environment
• LLM Server

So

• When you run Claude Code on your local MacBook, the first two are always local. The third is usually the Claude.ai server.
• When you ssh to a virtual private server (VPS) and install Claude Code there, the first two are your remote server. The third is still the Claude.ai server.
• When you run Claude RC on your virtual private server and code from your iPad using the Claude app, the end-user UI is on your iPad, the agent runtime environment is on your VPS, and the server is still Claude.ai.

Most people physically separate their tenancy, such as Claude Code, from their personal vs. work laptops. So in most cases, it's not a big deal.

But when you need multi-tenancy, it becomes super stressful. For example, say you have two different toolkits:

• personal toolkits (personal Notion, personal Sentry, personal Linear)
• workplace toolkits (company Notion, company Sentry, company Linear)

Most MCP auth states or code harnesses don't support profiles, so you can only log in to one.

So therefore... a natural evolution was to have both:

• a personal VPS with all personal toolkits set up
• a workplace VPS with all workspace toolkits set up

to physically isolate tenancies.

Now we've solved the multiple-profile issue, but the client's problems persist. Now let's get back to the environments:

• End-user UI
• Agent Runtime Environment
• LLM Server

All MCP auth or toolkit auth info should always be saved in the Agent Runtime Environment IMHO. However, a surprising number of harnesses tie them to the LLM server (such as Codex Apps or Claude.ai Plugins) or put them in the end-user UI (Claude Desktop or Codex Desktop).

Now the problem is:

• If the auth data is put on the LLM server, you cannot reuse LLM accounts across tenants
• If the auth data is put on the end-user UI, you cannot use the same app to access multi-tenants.

The only way to reliably isolate different auth information is thus:

• You ssh to a virtual private server (VPS) and run Claude Code there. Never use LLM server plugins.

Then

• End-user UI
• Agent Runtime Environment

are both isolated VPS, and

• LLM Server holds no information on the tenancy

This way, you can provide different toolkits, creating multiple dev environments.

• Letter to Mr. Matt Rickard on 2022-10-03

References

Screenshots as the Universal API

• With ML advancements, screenshots are now a universal data format.
  • (decoder) relatively easy to extract...
    • meaning (image-to-text)
    • layout information (object recognition)
    • text (OCR)
    • other metadata (formatting, fonts, etc.)
  • (encoder) diffusion-based models like Stable Diffusion and DALL-E (text-to-image) Prompt Engineering
• What's good:
  • Easier to parse than highly complex layout formats
    • No need to understand PDF data format
    • No need to hydrate webs for web crawlers
  • Universally available, easily copyable
    • Images aren't the most efficient encoding method for text.
    • But they can be the simplest for humans
    • You can copy objects from photos in the latest Apple iOS 16 update.
• Permissionless.
  • Many applications won't allow you to export data.
  • Screenshots are always available.
  • Related to when Naver Vibe attempted to steal other music players' market cap with Screenshot Recognition technology.
    • [튜토리얼] 다른 뮤직앱 플레이리스트, 쉽게 가져오는 법
    • '타사 음원 리스트 수초만에 이동' 네이버 바이브에 OCR 적용 - 전자신문
• More complex metadata
  • Look how effective image search is on mobile. Dogs, City, Oceans...
  • Some come from the actual image metadata, and others are inferred with On-device models.
  • Automatically encoding this data in traditional formats like PDF takes much longer.
• I wrote a reply like the following. Letter to Mr. Matt Rickard on 2022-10-03

Rethinking the PDF

• It's founder, John Warnock (co-founder of Adobe), prototyped a compatibility layer where documents would look and, most importantly, print (!) the same regardless of the computer they were viewed on (1993 video). This is the PDF.
• The "killer app" for PDF was tax returns - the IRS adopted PDF in 1996 because of a rumored frustration with the US Postal Service.
• Things that lack:
  • Enterprise-grade OCR for PDF documents still doesn't exist in 2022, albeit having state-of-the-art computer vision techniques.
  • Interactive and web-enabled forms. Sometimes it saves without the data filled in
  • Slow page loads. Better alternatives. EPUB, MOBI for texts. For generic use cases, DjVu.

References

• PDF processing and analysis with open-source tools