Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

42
LINES

< > BotCompany Repo | #1022891 // Input Tokenizer v2 [with colored display, OK]

JavaX source code (Dynamic Module) [tags: use-pretranspiled] - run with: Stefan's OS

Uses 911K of libraries. Click here for Pure Java version (11416L/62K).

!7

set flag DynModule.

cmodule InputTokenizer > DynBigInputAndSections {
  S functions = functions_default;
  sS functions_default = joinWithComma(
    f javaTok,
    f javaTokWithBrackets,
    f javaTokNPunct,
    f javaTokWithAngleBrackets);
    
  JComponent wrapTextArea(JTextArea textArea) {
    ret centerAndSouthWithMargins(textArea,
      centerAndEastWithMargin(
        withLabel("Tokenization functions:", dm_textField('functions)), jbutton("Restore default", r { setField(functions := functions_default) })));
  }
    
  void makeSections_impl() {
    S text = this.text;
    JTextPane tp = fontSizePlus(5, setFont(typeWriterFont(), jTextPane()));
    bool first = true;
    LPair<S, LS> toks = map_pcall(functions(), fn -> pair(fn, (LS) callAndMake(fn, text)));
    
    // find out max width
    int maxWidth = 0;
    for (LS tok : pairsB(toks)) {
      ai_tokenizationToTextPane(jTextPane(), tok);
      maxWidth = max(maxWidth, ai_tokenizationToTextPane_charsWritten!);
    }
    
    for (Pair<S, LS> fAndTok : toks) pcall {
      if (!first) appendToTextPane(tp, "\n"); else first = false;
      ai_tokenizationToTextPane(tp, fAndTok.b);
      appendToTextPane(tp, spaces(maxWidth-ai_tokenizationToTextPane_charsWritten!+1) + "[" + fAndTok.a + "]");
    }
    
    addSection("Tokenizations", tp);
  }
  
  Collection<S> functions() { ret tok_allIdentifiers(functions); }
}

Author comment

Began life as a copy of #1022887

download  show line numbers  debug dex  old transpilations   

Travelled to 7 computer(s): bhatertpkbcr, cfunsshuasjs, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1022891
Snippet name: Input Tokenizer v2 [with colored display, OK]
Eternal ID of this version: #1022891/13
Text MD5: b5b19e1789c11117c18dc5f2bee2d47b
Transpilation MD5: e2455c0e3e35ae88153e29d667f35326
Author: stefan
Category: javax / stefan's os / a.i.
Type: JavaX source code (Dynamic Module)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2019-04-14 19:14:51
Source code size: 1434 bytes / 42 lines
Pitched / IR pitched: No / No
Views / Downloads: 237 / 1254
Version history: 12 change(s)
Referenced in: [show references]