Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

34
LINES

< > BotCompany Repo | #2000483 // loadPage

New Tinybrain snippet

  public static String loadPage(String url) throws IOException {
    return loadPage(new URL(url));
  }
  
  public static String loadPage(URL url) throws IOException {
    System.out.println("Loading: " + url.toExternalForm());
    URLConnection con = url.openConnection();
    return loadPage(con, url);
  }

  public static String loadPage(URLConnection con, URL url) throws IOException {
    String contentType = con.getContentType();
    if (contentType == null)
      throw new IOException("Page could not be read: " + url);
    //Log.info("Content-Type: " + contentType);
    String charset = loadPage_guessCharset(contentType);
    Reader r = new InputStreamReader(con.getInputStream(), charset);
    StringBuilder buf = new StringBuilder();
    while (true) {
      int ch = r.read();
      if (ch < 0)
        break;
      //Log.info("Chars read: " + buf.length());
      buf.append((char) ch);
    }
    return buf.toString();
  }
  
  static String loadPage_guessCharset(String contentType) {
    Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*");
    Matcher m = p.matcher(contentType);
    /* If Content-Type doesn't match this pre-conception, choose default and hope for the best. */
    return m.matches() ? m.group(1) : "ISO-8859-1";
  }

Author comment

Began life as a copy of #2000482

download  show line numbers   

Snippet is not live.

Travelled to 12 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #2000483
Snippet name: loadPage
Eternal ID of this version: #2000483/1
Text MD5: 9dcfb95039bfc7840ec3ac89e0e45f7d
Author: stefan
Category:
Type: New Tinybrain snippet
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2015-08-02 14:54:05
Source code size: 1306 bytes / 34 lines
Pitched / IR pitched: No / Yes
Views / Downloads: 518 / 138
Referenced in: [show references]