Tok - string with tokenization (lazily computed) [1026190]

< > BotCompany Repo | #1026190 // Tok - string with tokenization (lazily computed)

JavaX fragment (include) [tags: use-pretranspiled]

sclass Tok {
  // either one can be null (bot not both)
  S s;    // joined
  LS tok; // tokenized
  
  ITokenizer tokenizer; // optional
  
  *() {}
  *(LS *tok) {}
  
  // a lambdaMapLike constructor - that's new
  // use like: Tok javaTok("bla")
  lambdaMapLike Tok(ITokenizer tokenizer, S s) {
    this.tokenizer = tokenizer;
    this.s = s;
  }
  
  S text() {
    if (s == null) s = join(tok);
    ret s;
  }
  
  LS tok() {
    if (tok == null) tok = tokenizer.tokenize(s);
    ret tok;
  }
  
  // Note: toString can trigger tokenization
  toString { ret appendSquareBracketed(text(), nTokens(countTokens(tok()))); }
}

Travelled to 6 computer(s): bhatertpkbcr, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt

Snippet ID:

#1026190

Snippet name:

Tok - string with tokenization (lazily computed)

Eternal ID of this version:

#1026190/11

Text MD5:

f77a0a561a0ca0d6dd36749857c8b727

Transpilation MD5:

9485032fdbc5c36150b440337a1285c9

Author:

stefan

Category:

javax / a.i.

Type:

JavaX fragment (include)

Public (visible to everyone):

Yes

Archived (hidden from active list):

Created/modified:

2019-11-28 14:00:07

Source code size:

655 bytes / 30 lines

Pitched / IR pitched:

No / No

Views / Downloads:

495 / 922

Version history:

10 change(s)

Referenced in:

[show references]

1	sclass Tok {
2	// either one can be null (bot not both)
3	S s; // joined
4	LS tok; // tokenized
5
6	ITokenizer tokenizer; // optional
7
8	*() {}
9	(LS tok) {}
10
11	// a lambdaMapLike constructor - that's new
12	// use like: Tok javaTok("bla")
13	lambdaMapLike Tok(ITokenizer tokenizer, S s) {
14	this.tokenizer = tokenizer;
15	this.s = s;
16	}
17
18	S text() {
19	if (s == null) s = join(tok);
20	ret s;
21	}
22
23	LS tok() {
24	if (tok == null) tok = tokenizer.tokenize(s);
25	ret tok;
26	}
27
28	// Note: toString can trigger tokenization
29	toString { ret appendSquareBracketed(text(), nTokens(countTokens(tok()))); }
30	}