Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

30
LINES

< > BotCompany Repo | #1026190 // Tok - string with tokenization (lazily computed)

JavaX fragment (include) [tags: use-pretranspiled]

Libraryless. Click here for Pure Java version (2324L/15K).

sclass Tok {
  // either one can be null (bot not both)
  S s;    // joined
  LS tok; // tokenized
  
  ITokenizer tokenizer; // optional
  
  *() {}
  *(LS *tok) {}
  
  // a lambdaMapLike constructor - that's new
  // use like: Tok javaTok("bla")
  lambdaMapLike Tok(ITokenizer tokenizer, S s) {
    this.tokenizer = tokenizer;
    this.s = s;
  }
  
  S text() {
    if (s == null) s = join(tok);
    ret s;
  }
  
  LS tok() {
    if (tok == null) tok = tokenizer.tokenize(s);
    ret tok;
  }
  
  // Note: toString can trigger tokenization
  toString { ret appendSquareBracketed(text(), nTokens(countTokens(tok()))); }
}

download  show line numbers  debug dex  old transpilations   

Travelled to 6 computer(s): bhatertpkbcr, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1026190
Snippet name: Tok - string with tokenization (lazily computed)
Eternal ID of this version: #1026190/11
Text MD5: f77a0a561a0ca0d6dd36749857c8b727
Transpilation MD5: 9485032fdbc5c36150b440337a1285c9
Author: stefan
Category: javax / a.i.
Type: JavaX fragment (include)
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2019-11-28 14:00:07
Source code size: 655 bytes / 30 lines
Pitched / IR pitched: No / No
Views / Downloads: 182 / 537
Version history: 10 change(s)
Referenced in: [show references]