OpenClaw, Hermes, Poke를 쓰며 느낀 단점들

내가 만약 워크스페이스 에이전트를 만든다면 어떻게 만들까?

메모리

• 육하원칙에 맞게 기억해야하는데, 이걸 잘 못 함
  • 맥락을 종종 헷갈려함
• 메모리 툴을 명시적으로 불러야 함
  • 이는 Hermes에서 많이 완화된 부분

Slack

• 답장을 매번 하려 함. 쓰레드에 달리는 모든 글에 답을 함.
  • 매번 답할 필요는 없음. 필요할 때만 하면 됨.
• 언급하지 않은 메시지는 읽지 않음.
  • 모든 워크스페이스의 변화를 계속 보고 있어야 함, 그리고 기억해야 함

• Letter to Mr. Matt Rickard on 2022-10-03

References

Screenshots as the Universal API

• With ML advancements, screenshots are now a universal data format.
  • (decoder) relatively easy to extract...
    • meaning (image-to-text)
    • layout information (object recognition)
    • text (OCR)
    • other metadata (formatting, fonts, etc.)
  • (encoder) diffusion-based models like Stable Diffusion and DALL-E (text-to-image) Prompt Engineering
• What's good:
  • Easier to parse than highly complex layout formats
    • No need to understand PDF data format
    • No need to hydrate webs for web crawlers
  • Universally available, easily copyable
    • Images aren't the most efficient encoding method for text.
    • But they can be the simplest for humans
    • You can copy objects from photos in the latest Apple iOS 16 update.
• Permissionless.
  • Many applications won't allow you to export data.
  • Screenshots are always available.
  • Related to when Naver Vibe attempted to steal other music players' market cap with Screenshot Recognition technology.
    • [튜토리얼] 다른 뮤직앱 플레이리스트, 쉽게 가져오는 법
    • '타사 음원 리스트 수초만에 이동' 네이버 바이브에 OCR 적용 - 전자신문
• More complex metadata
  • Look how effective image search is on mobile. Dogs, City, Oceans...
  • Some come from the actual image metadata, and others are inferred with On-device models.
  • Automatically encoding this data in traditional formats like PDF takes much longer.
• I wrote a reply like the following. Letter to Mr. Matt Rickard on 2022-10-03

Rethinking the PDF

• It's founder, John Warnock (co-founder of Adobe), prototyped a compatibility layer where documents would look and, most importantly, print (!) the same regardless of the computer they were viewed on (1993 video). This is the PDF.
• The "killer app" for PDF was tax returns - the IRS adopted PDF in 1996 because of a rumored frustration with the US Postal Service.
• Things that lack:
  • Enterprise-grade OCR for PDF documents still doesn't exist in 2022, albeit having state-of-the-art computer vision techniques.
  • Interactive and web-enabled forms. Sometimes it saves without the data filled in
  • Slow page loads. Better alternatives. EPUB, MOBI for texts. For generic use cases, DjVu.

References

• PDF processing and analysis with open-source tools