Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

29
LINES

< > BotCompany Repo | #1022890 // ai_tokenizationToTextPane - visualize C and N tokens differently

JavaX fragment (include)

static new ThreadLocal<Int> ai_tokenizationToTextPane_charsWritten;

static JTextPane ai_tokenizationToTextPane(LS tok) {
  ret ai_tokenizationToTextPane(fontSizePlus(7, jtextpane()), tok);
}

static JTextPane ai_tokenizationToTextPane(JTextPane tp, LS tok) {
  bool spaced = true;
  int chars = 0;
  for i over tok: {
    if (!spaced) {
      appendToTextPane(tp, " ");
      ++chars;
      spaced = true;
    }
    S t = tok.get(i);
    if (empty(t)) continue;
    SimpleAttributeSet style;
    if (odd(i)) // C token
      style = style_background(Color.yellow, style_foreground(Color.black));
    else // N token (whitespace)
      style = style_foreground(Color.gray, style_background(colorFromHex("#E0E0E0")));
    appendToTextPane(tp, t, style);
    chars += l(t);
    spaced = false;
  }
  setTL(ai_tokenizationToTextPane_charsWritten, chars);
  ret tp;
}

download  show line numbers  debug dex  old transpilations   

Travelled to 7 computer(s): bhatertpkbcr, cfunsshuasjs, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1022890
Snippet name: ai_tokenizationToTextPane - visualize C and N tokens differently
Eternal ID of this version: #1022890/10
Text MD5: 38e806367dfe6ee9982713e0a035d262
Author: stefan
Category: javax / gui
Type: JavaX fragment (include)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2019-04-14 18:27:42
Source code size: 891 bytes / 29 lines
Pitched / IR pitched: No / No
Views / Downloads: 190 / 235
Version history: 9 change(s)
Referenced in: [show references]