Hugging Face's logo

Spaces:
ggunio
/
intelligent-tokenizer-v6-demo
Sleeping

App Files Community
intelligent-tokenizer-v6-demo / app.py

Commit History

Fix: Correct embedding dimension (1280-dim, not 768-dim)
3a4dfa2
verified

ggunio commited on

Fix: Extend max_length for proper text reconstruction
85b3a58
verified

ggunio commited on

Final update: Complete documentation on semantic boundaries, better examples, clear explanation of the value proposition
2068c6b
verified

ggunio commited on

Fix: Correct embedding count calculation, add full documentation and explanations
77a029c
verified

ggunio commited on

Simplify interface: focus on reconstruction test with embedding count and compression stats
8ba68ce
verified

ggunio commited on

Upload app.py with huggingface_hub
4e3eeae
verified

ggunio commited on

Add core purpose explanation to demo
b1a9b59

ggunio Claude commited on

Rollback to stable version and add demo limitation notice
50ce9e8

ggunio Claude commited on

Update demo with sliding window implementation
ee44b85

ggunio Claude commited on

Add detailed explanation of chunking effects on token boundaries
0a70666

ggunio commited on

Fix UTF-8 safe chunking, token boundary visualization, and embedding display
0250815

ggunio commited on

Fix PyTorch 2.6 weights_only error
83b0066

ggunio commited on

Update app.py
e0f2654
verified

ggunio commited on

Fix reconstruction issue by downloading checkpoint from HF model repo
bb03134

ggunio commited on

Fix import error by adding core module files
c2e3f6e

ggunio commited on

Update to B2NL v6.1.2 POC - 18.6:1 compression with 6 languages (Korean, English, Chinese, Japanese, Spanish, Arabic)
13c2c77

ggunio commited on

Update to B2NL v6.1.1 - 97.71% reconstruction achieved!
2607a65
verified

ggunio commited on

Upload app.py with huggingface_hub
b9a9591
verified

ggunio commited on

Upload app.py with huggingface_hub
7b4dd01
verified

ggunio commited on

Upload app.py with huggingface_hub
41b2d12
verified

ggunio commited on

Upload app.py with huggingface_hub
424a951
verified

ggunio commited on