研究・調査 RESEARCH
AskDona / GFLOPSが公開する長尺レポート・ベンチマークです。 Long-form research and benchmarks published by AskDona / GFLOPS.
リサーチ Research
Is SubQ Snap deterministic? We sent it the same request 30 times. Is SubQ Snap deterministic? We sent it the same request 30 times.
Byte-identical requests, no label flips, and probabilities that depend on whether the calls were sequential or concurrent. The same test we ran on Jev two weeks ago. Byte-identical requests, no label flips, and probabilities that depend on whether the calls were sequential or concurrent. The same test we ran on Jev two weeks ago.
Is Jev deterministic? We sent it the same request 30 times. Is Jev deterministic? We sent it the same request 30 times.
Byte-identical requests, a pinned model version, and two flipped labels. Why that matters for regulated assessment, and why we are making our whole pipeline deterministic. Byte-identical requests, a pinned model version, and two flipped labels. Why that matters for regulated assessment, and why we are making our whole pipeline deterministic.
LLMを用いたクエリ分類を実務で行うには：AskDonaチャットボットの数万件のクエリ分析から得られた実践的知見 LLM-based Query Classification Practices: Lessons from Analyzing Tens of Thousands of AskDona Chatbot Queries
本稿では、理化学研究所 計算科学研究センター（R-CCS）が運用する「富岳サポートサイト」および「スパコン成果ナビ」という2つの実稼働チャットボットを題材とし、LLMを用いたクエリ分類をどのように実施し、分類結果の品質を確保したのか、その具体的な手順と実施上の工夫を解説します。 This article uses two production chatbots operated by the RIKEN Center for Computational Science (R-CCS), the Fugaku Support Site and the Supercomputer Report Navigator, to explain how LLM-based query classification was implemented and how the quality of the resulting labels was maintained.
本番RAGにおけるStructural Blind Spots Structural Blind Spots in Production RAG
「精度N%」だけでは、本番RAGパイプラインの最も重要な評価軸——クエリクラスの構造的カバレッジ——を捉え損ねる。AskDonaの本番運用から得たStructural Blind Spotsの概略を示す。 Why accuracy alone cannot evaluate RAG pipelines — an outline of Structural Blind Spots, the class-level capability that 'N% accuracy' misses, drawn from AskDona production deployments at RIKEN R-CCS.