Important: This documentation covers Yarn 1 (Classic).
For Yarn 2+ docs and migration guide, see yarnpkg.com.

Package detail

htp

YounGoat2.1kMIT0.13.0

Easy http/https client

http, https

readme

htp

Maybe the easiest but still strong http client you have ever meet.

coverage status of github.com/YounGoat/nodejs.htp

total downloads of htp htp's License latest version of htp coverage status of github.com/YounGoat/nodejs.htp dependencies of github.com/YounGoat/nodejs.htp devDependencies of github.com/YounGoat/nodejs.htp build status of github.com/YounGoat/nodejs.htp star github.com/YounGoat/nodejs.htp

Languages / 简体中文
If links in this document not avaiable, please access README on GitHub directly.

Table of Contents

Description

Another choice for you to finish HTTP or HTTPS request.

htp.logo

ToC

Get Started

var htp = requrie('htp');
// OR, since v0.3.0, an alias "usa" is available.
var htp = require('usa');

// GET & callback
htp.get('http://www.example.com/', function(err, response) {
    if (err) {
        // Exception throwed on requesting.
    }
    else {
        // Response received.
        response.statusCode;
        response.statusMessage;
        response.httpVersion;
        response.headers;
        response.body;
        response.bodyBuffer;
        response.bodyDecompressed;
        response.performance;
    }
});

// POST & promise
var data = { username: 'youngoat', password: 'helloworld' };
htp.post('http://www.example.com/login', data).then(function(response) {
    // ...
}).catch(function(err) {
    // ...
});

// Customized settings.
var client = new htp({
    response_timeout: 1000
});
client.request('GET', 'http://www.example.com/', function(err, response) {
    // ...
});

API

The Response Object

If request achieved successfully, a Response object will be passed. The Response object SHOULD contain the following properties:

  • response.statusCode number
  • response.statusMessage string
  • response.httpVersion string
  • response.headers Object
  • response.body string | Object(json) | null
  • response.bodyBuffer Buffer | null
  • response.bodyDecompressed
  • response.performance Object

Basic API

// To execute request with default settings.
htp(
    /*string*/ REQUSET_METHOD_NAME,
    /*string*/ URL,
    /*OPTIONAL object*/ HEADERS,
    /*OPTIONAL string | object | stream.Readable*/ BODY,
    /*OPTIONAL function*/ CALLBACK
);
  • HEADERS, BODY and CALLBACK are all optional.
  • htp returns undefined while CALLBACK offered, otherwise a promise will be returned.
  • htp distinguishes arguments by their types and order. However, it may be ambiguous if there is one, but only one object argument offered. What is it regarded as, HEADERS or BODY? If the method is defined with explicit payload, the object will be regarded as BODY, otherwise it will be regarded as HEADERS. See methods-without-payloads.js for details.

Another style maybe many coders prefer to is htp.<lowercase_method_name>( /* ... */ ), e.g.

htp.get('http://www.example.com/', function(error, response) {
    // ...
});

Without CALLBACK offered, htp will return a promise.

htp.get('http://www.example.com/')
    .then(function(response) { /* ... */ })
    .catch(function(error) { /* ... */ })
    ;

Piping API

Since v0.1.0, a streamable subset htp.piping is available. Whether or not CALLBACK offered, it will always return a readable stream.

htp.piping
    .get('http://download.example.com/data.json')
    .pipe(fs.createWriteStream('data.json'))
    ;

// A property function named with "piping" prefixed (in camelCase) is equivalent.
htp
    .pipingGet('http://download.example.com/data.json')
    .pipe(fs.createWriteStream('data.json'))
    ;

The stream returned by htp.piping.<method>() may emit following events:

  • Event: 'dns'
    • { address string, family number }
  • Event: 'connect'
  • Event: 'response'
    • response Object
      Along with argument response which is a subset of the final response object.
  • events which a readable stream may emit
    See Class: stream.Readable for details.

ATTENTION: Which returned by htp.piping.<method>() and then returned by .pipe() are not the same stream.

Advanced API

// Create a customized user-agent.
var request = new htp({
    hostname: 'www.example.com',
});

request.get('/index.html', function(err, response) {
    // ...
});

Here are options available when creating a customized user agent:

  • options.protocol ENUM('http', 'https')
    Default protocol.

  • options.hostname string
    Default hostname (port excluded).

  • options.port number
    Default port.

  • options.piping boolean
    If set true, a readable stream will be returned whether or not CALLBACK is present. Otherwise, a promise will be returned when CALLBACK is absent.

  • options.pipingOnly boolean
    Only effective in piping mode. If set true, reponse data will no longer be staged and returned, and argument response passed to CALLBACK will no longer have properties { body, bodyBuffer, bodyDcompressed }. You can only obtain response data through pipe.

  • options.proxy string
    Proxy, e.g. "http://localhost:8080/".

  • options.request_timeout number (unit: ms)
    Max time to finish the whole request.

  • options.dns_timeout number (unit: ms)
    Max time to resolve hostname.

  • options.dns_ttl number (unit: seconds)
    Time-to-live of DNS resolving result.

  • options.dnsAgent dns-agent
    An instance of dns-agent.

  • options.plugin_timeout number (unit: ms)
    Max time to plug into socket.

  • options.connect_timeout number (unit: ms)
    Max time to shake-hands with target server.

  • options.response_timeout number (unit: ms)
    Max time to recieve the first response from target server.

  • options.chunk_timeout number (unit: ms)
    Max time two data chunks.

  • options.data_timeout number (unit: ms)
    Max time to receive all data.

Some options from tls.connect() are also accepted and will be applied on HTTPs requests:

  • options.rejectUnauthorized boolean

See settings.js for default values of options.

Class SimpleAgent

By creating an instance of SimpleAgent, developers are able to create a customized and resuable htp.

var Agent = require('htp/SimpleAgent');

// Create an instance.
var agent = new Agent({
    endPoint: 'http://www.example.com/'
});

var p = agent.get('/index.html');
p.then(function(bodyBuffer) {
    console.log(bodyBuffer.toString('utf8'));
}).catch(function(err) {
    // ...
});

Acceptable options accepted by htp/SimpleAgent are:

  • Function beforeRequest({ method, url, headers, body, callback })
    If offered, this function will be invoked with an object passed in before real HTTP request starts. The passed in object is made up five properties. The function SHOULD return void or an object made up all or some of the properties. If an object returned, the properties will be used on requesting.
  • Function beforeCallback(error, data)
    If offered, this function will be invoked before callback invoked. The returned value will be used as real data and passed to callback(null, data) or resolve(data).
  • string endPoint
  • object headers
  • object query
  • object settings
    Settings used to customise the user-agent. See Advanced API.

Timeline

Here is the timeline of an htp request:
HTP request process

About

For convenience, this package has following names (alias):

  • htp
    Because the main business of this package is to execute HTTP/HTTPS requests, I named it htp which looks like http with double t replaced by one.

  • usa
    It is not the United States of America. usa is abbreviation of USerAgent.

Recommendations

changelog

htp

Notable changes to this project will be documented in this file. This project adheres to Semantic Versioning 2.0.0.

[0.13.0] - 2021-05-13

  • 不再自动按 HTTP method 名称创建 htp.piping<Method>htp.pipingOnly<Method> 方法,但仍支持按 htp.piping.<method>htp.pipingOnly.<method> 方式引用。

    [0.12.1] - 2021-04-27

  • 取消默认字符集为 utf8 的设置。

  • 当内容类型(content-type)未指定字符集时,根据类型数据库提供的默认字符集(如有),尝试将其转换为字符串。

[0.12.0] - Dec 18th, 2018

  • Support proxy!

[0.11.0] - Nov 19th, 2018

  • Fixed the bug trying gunzip a HEAD response.
  • Response object property network added.

[0.10.4] - Aug 16th, 2018

  • Accept instance of Array as BODY.

[0.10.3] - Aug 7th, 2018

  • An instance of DnsAgent may be passed in via htp({ dnsAgent }) to create a customised http user agent.
  • Fixed the bug that dnsAgent and http(s)Agent unnecessarily created when no special settings property passed in.

[0.10.2] - Aug 5th, 2018

  • Add mothod COPY to the method-without-payload list.
  • Allow all methods to request with payload as long as no arguments omitted. E.g.
    // An OVERLOAD2.UnmatchingException will be thrown before.
    // However, no exception thrown now.
    htp.delete('http://example.com/source.html', null, { Backup: '/target.html' });

[0.10.1] - Aug 3rd, 2018

  • Fixed the bug that the package depends on itself.

[0.10.0] - Aug 1st, 2018

  • Fixed the bug that when performance data sometimes incomplete when passed-in agent enables keepAlive option:

    // Event `connect` will be not triggered when `socket` is re-used and already connected.
    socket.once('connect', function() {
       timeout.end('CONNECT');
       emitOnBodyStream('connect');
    
       timeout.start('RESPONSE', fnDone);
       timeout.start('DATA', fnDone);
       Object.assign(connection, object2.clone(socket, [ /^remote/, /^local/ ]));
    });
  • Fixed the bug that customised settings keepAlive not effective.

  • Change default settings keepAlive to true.

[0.9.0] - June 19th, 2018

  • Accept all instance of stream rather than only those of stream.Readable as body.
  • Support charset 'ISO-8859-1' on parsing response body.
  • Delete items with undefined value from headers. Otherwise, an Error will be thrown when http/https try to setHeader with undefined value.

[0.8.2] - May 10th, 2018

  • Catch error on processing response encoded via gzip/deflate.

[0.8.1] - May 9th, 2018

  • Fixed the bug that dependencies missed.

[0.8.0] - Apr 24th, 2018

  • Default settings changed on response.timeout and data.timeout.

[0.7.1] - April 1st, 2018

  • Fixed the bug that on response with unsupported charset, an unhandled exception will be thrown.

[0.7.0] - Mar 23rd, 2018 - RISKY

  • http.request() or https.request() is wrapped with try ... catch ....
  • Standlone instances of http.Agent and https.Agent are used in replacement of http.globalAgent and https.globalAgent.
  • Default setting dns_ttl is changed to 60 (in seconds) from 600000 (in milli-seconds). And the unit of options.dns_ttl is changed to seconds.

[0.6.1] - Feb 7th, 2018 - RISKY

Changed, RISKY

  • When instance of stream.Readable used as HTTP body, errors on the stream will be caught and the request action will be ceased forcely.

[0.6.0] - Feb 1st, 2018

New

  • Property beforeCallback added to options argument in new SimpleAgent(options).

[0.5.1] - Jan 16, 2018

Changed, RISKY

  • In this version, hostname in the URL will be replaced with IP address returned by dns-agent before being passed to http.request() or https.request() to avoid repetitive DNS requests for resolving the same hostname.
    在这个版本中,我们用 dns-agent 返回的 IP 地址替换 URL 中的主机名,然后才将其传递至 http.request()https.request() 方法。dns-agent 本身具备可靠的及可调节的缓存功能,如果针对同一主机名的 URL 地址,发生高并发调用或短时间内的大量调用,这一作法可以有效避免系统重复发起 DNS 解析。

[0.5.0] - Jan 8, 2018

Fixed, IMPORTANT

  • Fixed the bug that when invoking methods with payload (e.g. POST and PUT) of instanceof of htp/SimpleAgent, body of falsed value 0-length string will be ignored.
  • Force the clientRequest to abort and the incommingMessage (response) to destroy on error ocurring.
    此前,当错误发生后,请求和响应可能并未终止,由此可能导致不可预知的问题。

[0.4.2] - Jan 2, 2018

  • On piping mode, pass address object to listener of event dns.

[0.4.1] - Jan 2, 2018

Fixed

  • Fixed the bug that, on piping mode, event dns is missed if hostname has been resolved and cached before.
  • Fixed the bug that, on piping mode, event error is missed.

[0.4.0] - Dec 13, 2017

  • dns-agent is depended while htp is resolving domain names.

[0.3.0] - Dec 8, 2017

  • Setting rejectUnauthorized added.

[0.2.0]

New

  • Byeond basic and piping mode, new pipingOnly mode is offered.
  • Returned stream in piping mode will emit events and may be catched via on(<eventName>, ...) method.
  • README in Simplifed Chinese language.

Fixed

  • In previous version, htp/SimpleAgent will invoke callback() (if passed) twice if there is some exception throwed in callback() itself.

What is difference between piping and pipingOnly ?

In piping mode, you may get data via piping to a writable stream, or reading response.body and response.bodyBuffer directly in function callback(err, response) which is passed as the last argument.

[0.1.1] - Nov 23, 2017

Fixed

  • In previous versions, htp cannot recognized some key fields in response headers if their names are not lowercased.

[0.1.0] - 2017-11-12

Milestone

Now, htp is streamable. E.g.

htp.piping.get('http://www.example.com/').pipe(fs.createWriteStream('index.html'));

[0.0.5] - 2017-11

htp/sSimpleAgent added.

[0.0.4] - 2017-10

Add new action property to Errors throwed. Value of the property is made up of the method name and the url joined with a whitespace, e.g.

htp.get('http://www.example.com/', function(err, response) {
    if (err) {
        err.action == 'GET http://www.example.com/'; // true
    }
});

[0.0.3] - 2017-09

No promise created or returned when callback() functions offered.

[0.0.2] - 2017-07, https

HTTPS requests are supported.

[0.0.1] - 2017-07

Released.


This CHANGELOG.md follows Keep a CHANGELOG.