History

Dušan Borovčanin 3d3aa525a6 NOISSUE - Switch to Google Zanzibar Access control approach (#1919 ) * Return Auth service Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update Compose to run with SpiceDB and Auth svc Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update auth gRPC API Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Remove Users' policies Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Move Groups to internal Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Use shared groups in Users Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Remove unused code Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Use pkg Groups in Things Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Remove Things groups Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Make imports consistent Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update Groups networking Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Remove things groups-specific API Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Move Things Clients to the root Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Move Clients to Users root Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Temporarily remove tracing Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Fix imports Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add buffer config for gRPC Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update auth type for Things Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Use Auth for login Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add temporary solution for refresh token Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update Tokenizer interface Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Updade tokens issuing Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Fix token issuing Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update JWT validator and refactor Tokenizer Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Rename access timeout Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Rename login to authenticate Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update Identify to use SubjectID Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add Auth to Groups Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Use the Auth service for Groups Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update auth schema Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Fix Auth for Groups Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add auth for addons (#14) Signed-off-by: Arvindh <arvindh91@gmail.com> Speparate Login and Refresh tokens Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Merge authN and authZ requests for things Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add connect and disconnect Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update sharing Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Fix policies addition and removal Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Update relation with roels Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Add gRPC to Things Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Assign and Unassign members to group and Listing of Group members (#15) * add auth for addons Signed-off-by: Arvindh <arvindh91@gmail.com> * add assign and unassign to group Signed-off-by: Arvindh <arvindh91@gmail.com> * add group incomplete repo implementation Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Move coap mqtt and ws policies to spicedb (#16) Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Remove old policies Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> NOISSUE - Things authorize to return thingID (#18) This commit modifies the authorize endpoint to the grpc endpoint to return thingID. The authorize endpoint allows adapters to get the publisher of the message. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Add Groups to users service (#17) * add assign and unassign to group Signed-off-by: Arvindh <arvindh91@gmail.com> * add group incomplete repo implementation Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users stable 1 Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users stable 2 Signed-off-by: Arvindh <arvindh91@gmail.com> * groups for users & things Signed-off-by: Arvindh <arvindh91@gmail.com> * Amend signature Signed-off-by: Arvindh <arvindh91@gmail.com> * fix merge error Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Fix es code (#21) Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Fix Bugs (#20) * fix bugs Signed-off-by: Arvindh <arvindh91@gmail.com> * fix bugs Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Test e2e (#19) * fix: connect method Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * fix: e2e Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * fix changes in sdk and e2e Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(docker): remove unnecessary port mapping Remove the port mapping for MQTT broker in the docker-compose.yml file. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * Enable group listing Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(responses): update ChannelsPage struct The ChannelsPage struct in the responses.go file has been updated. The "Channels" field has been renamed to "Groups" to provide more accurate naming. This change ensures consistency and clarity in the codebase. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(things): add UpdateClientSecret method Add the UpdateClientSecret method to the things service. This method allows updating the client secret for a specific client identified by the provided token, id, and key parameters. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> --------- Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Use smaller buffers for gRPC Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Clean up tests (#22) Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Add Connect Disconnect endpoints (#23) * fix bugs Signed-off-by: Arvindh <arvindh91@gmail.com> * fix bugs Signed-off-by: Arvindh <arvindh91@gmail.com> * fix list of things in a channel and Add connect disconnect endpoint Signed-off-by: Arvindh <arvindh91@gmail.com> * fix list of things in a channel and Add connect disconnect endpoint Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Add: Things share with users (#25) * fix list of things in a channel and Add connect disconnect endpoint Signed-off-by: Arvindh <arvindh91@gmail.com> * add: things share with other users Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Rename gRPC Services (#24) * Rename things and users auth service Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * docs: add authorization docs for gRPC services Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * Rename things and users grpc services Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * Remove mainflux.env package Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> --------- Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Add: Listing of things, channels, groups, users (#26) * add: listing of channels, users, groups, things Signed-off-by: Arvindh <arvindh91@gmail.com> * add: listing of channels, users, groups, things Signed-off-by: Arvindh <arvindh91@gmail.com> * add: listing of channels, users, groups, things Signed-off-by: Arvindh <arvindh91@gmail.com> * add: listing of channels, users, groups, things Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Clean Up Users (#27) * feat(groups): rename redis package to events - Renamed the `redis` package to `events` in the `internal/groups` directory. - Updated the file paths and names accordingly. - This change reflects the more accurate purpose of the package and improves code organization. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(auth): Modify identity method Change request and response of identity method Add accessToken and refreshToken to Token response Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * clean up users, remove dead code Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(users): add unit tests for user service This commit adds unit tests for the user service in the `users` package. The tests cover various scenarios and ensure the correct behavior of the service. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> --------- Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Add: List of user groups & removed repeating code in groups (#29) * removed repeating code in list groups Signed-off-by: Arvindh <arvindh91@gmail.com> * add: list of user group Signed-off-by: Arvindh <arvindh91@gmail.com> * fix: otel handler operator name for endpoints Signed-off-by: Arvindh <arvindh91@gmail.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Clean Up Things Service (#28) * Rework things service Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * add tests Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> --------- Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Clean Up Auth Service (#30) * clean up auth service Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> * feat(auth): remove unused import Remove the unused import of `emptypb` in `auth.pb.go`. This import is not being used in the codebase and can be safely removed. Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> --------- Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * NOISSUE - Update API docs (#31) Signed-off-by: rodneyosodo <blackd0t@protonmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Remove TODO comments and cleanup the code Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> * Update dependenices Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> --------- Signed-off-by: Arvindh <arvindh91@gmail.com> Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com> Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com> Signed-off-by: rodneyosodo <blackd0t@protonmail.com> Co-authored-by: b1ackd0t <28790446+rodneyosodo@users.noreply.github.com> Co-authored-by: Arvindh <30824765+arvindh123@users.noreply.github.com>		2023-10-15 22:02:13 +02:00
..
css	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
html	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
js	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
json	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
svg	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
xml	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
.gitattributes	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
.gitignore	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
.golangci.yml	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
Dockerfile	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
LICENSE	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
Makefile	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
README.md	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
common.go	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00
minify.go	NOISSUE - Switch to Google Zanzibar Access control approach (#1919 )	2023-10-15 22:02:13 +02:00

README.md

Minify

Online demo if you need to minify files now.

Binaries of CLI for various platforms. See CLI for more installation instructions.

Python bindings install with pip install tdewolff-minify

JavaScript bindings install with npm i @tdewolff/minify

.NET bindings install with Install-Package NMinify or dotnet add package NMinify, thanks to Jonas Kamsker for the port

Did you know that the shortest valid piece of HTML5 is <!doctype html><title>x</title>? See for yourself at the W3C Validator!

Minify is a minifier package written in Go. It provides HTML5, CSS3, JS, JSON, SVG and XML minifiers and an interface to implement any other minifier. Minification is the process of removing bytes from a file (such as whitespace) without changing its output and therefore shrinking its size and speeding up transmission over the internet and possibly parsing. The implemented minifiers are designed for high performance.

The core functionality associates mimetypes with minification functions, allowing embedded resources (like CSS or JS within HTML files) to be minified as well. Users can add new implementations that are triggered based on a mimetype (or pattern), or redirect to an external command (like ClosureCompiler, UglifyCSS, ...).

Roadmap

Use ASM/SSE to further speed-up core parts of the parsers/minifiers
Improve JS minifiers by shortening variables and proper semicolon omission
Speed-up SVG minifier, it is very slow
Proper parser error reporting and line number + column information
Generation of source maps (uncertain, might slow down parsers too much if it cannot run separately nicely)
Create a cmd to pack webfiles (much like webpack), ie. merging CSS and JS files, inlining small external files, minification and gzipping. This would work on HTML files.

Prologue

Minifiers or bindings to minifiers exist in almost all programming languages. Some implementations are merely using several regular expressions to trim whitespace and comments (even though regex for parsing HTML/XML is ill-advised, for a good read see Regular Expressions: Now You Have Two Problems). Some implementations are much more profound, such as the YUI Compressor and Google Closure Compiler for JS. As most existing implementations either use JavaScript, use regexes, and don't focus on performance, they are pretty slow.

This minifier proves to be that fast and extensive minifier that can handle HTML and any other filetype it may contain (CSS, JS, ...). It is usually orders of magnitude faster than existing minifiers.

Installation

Make sure you have Git and Go (1.18 or higher) installed, run

mkdir Project
cd Project
go mod init
go get -u github.com/tdewolff/minify/v2

Then add the following imports to be able to use the various minifiers

import (
	"github.com/tdewolff/minify/v2"
	"github.com/tdewolff/minify/v2/css"
	"github.com/tdewolff/minify/v2/html"
	"github.com/tdewolff/minify/v2/js"
	"github.com/tdewolff/minify/v2/json"
	"github.com/tdewolff/minify/v2/svg"
	"github.com/tdewolff/minify/v2/xml"
)

You can optionally run go mod tidy to clean up the go.mod and go.sum files.

See CLI tool for installation instructions of the binary.

Docker

If you want to use Docker, please see https://hub.docker.com/r/tdewolff/minify.

$ docker run -it tdewolff/minify --help

API stability

There is no guarantee for absolute stability, but I take issues and bugs seriously and don't take API changes lightly. The library will be maintained in a compatible way unless vital bugs prevent me from doing so. There has been one API change after v1 which added options support and I took the opportunity to push through some more API clean up as well. There are no plans whatsoever for future API changes.

Testing

For all subpackages and the imported parse package, test coverage of 100% is pursued. Besides full coverage, the minifiers are fuzz tested using github.com/dvyukov/go-fuzz, see the wiki for the most important bugs found by fuzz testing. These tests ensure that everything works as intended and that the code does not crash (whatever the input). If you still encounter a bug, please file a bug report!

Performance

The benchmarks directory contains a number of standardized samples used to compare performance between changes. To give an indication of the speed of this library, I've ran the tests on my Thinkpad T460 (i5-6300U quad-core 2.4GHz running Arch Linux) using Go 1.15.

name                              time/op
CSS/sample_bootstrap.css-4          2.70ms ± 0%
CSS/sample_gumby.css-4              3.57ms ± 0%
CSS/sample_fontawesome.css-4         767µs ± 0%
CSS/sample_normalize.css-4          85.5µs ± 0%
HTML/sample_amazon.html-4           15.2ms ± 0%
HTML/sample_bbc.html-4              3.90ms ± 0%
HTML/sample_blogpost.html-4          420µs ± 0%
HTML/sample_es6.html-4              15.6ms ± 0%
HTML/sample_stackoverflow.html-4    3.73ms ± 0%
HTML/sample_wikipedia.html-4        6.60ms ± 0%
JS/sample_ace.js-4                  28.7ms ± 0%
JS/sample_dot.js-4                   357µs ± 0%
JS/sample_jquery.js-4               10.0ms ± 0%
JS/sample_jqueryui.js-4             20.4ms ± 0%
JS/sample_moment.js-4               3.47ms ± 0%
JSON/sample_large.json-4            3.25ms ± 0%
JSON/sample_testsuite.json-4        1.74ms ± 0%
JSON/sample_twitter.json-4          24.2µs ± 0%
SVG/sample_arctic.svg-4             34.7ms ± 0%
SVG/sample_gopher.svg-4              307µs ± 0%
SVG/sample_usa.svg-4                57.4ms ± 0%
SVG/sample_car.svg-4                18.0ms ± 0%
SVG/sample_tiger.svg-4              5.61ms ± 0%
XML/sample_books.xml-4              54.7µs ± 0%
XML/sample_catalog.xml-4            33.0µs ± 0%
XML/sample_omg.xml-4                7.17ms ± 0%

name                              speed
CSS/sample_bootstrap.css-4        50.7MB/s ± 0%
CSS/sample_gumby.css-4            52.1MB/s ± 0%
CSS/sample_fontawesome.css-4      61.2MB/s ± 0%
CSS/sample_normalize.css-4        70.8MB/s ± 0%
HTML/sample_amazon.html-4         31.1MB/s ± 0%
HTML/sample_bbc.html-4            29.5MB/s ± 0%
HTML/sample_blogpost.html-4       49.8MB/s ± 0%
HTML/sample_es6.html-4            65.6MB/s ± 0%
HTML/sample_stackoverflow.html-4  55.0MB/s ± 0%
HTML/sample_wikipedia.html-4      67.5MB/s ± 0%
JS/sample_ace.js-4                22.4MB/s ± 0%
JS/sample_dot.js-4                14.5MB/s ± 0%
JS/sample_jquery.js-4             24.8MB/s ± 0%
JS/sample_jqueryui.js-4           23.0MB/s ± 0%
JS/sample_moment.js-4             28.6MB/s ± 0%
JSON/sample_large.json-4           234MB/s ± 0%
JSON/sample_testsuite.json-4       394MB/s ± 0%
JSON/sample_twitter.json-4        63.0MB/s ± 0%
SVG/sample_arctic.svg-4           42.4MB/s ± 0%
SVG/sample_gopher.svg-4           19.0MB/s ± 0%
SVG/sample_usa.svg-4              17.8MB/s ± 0%
SVG/sample_car.svg-4              29.3MB/s ± 0%
SVG/sample_tiger.svg-4            12.2MB/s ± 0%
XML/sample_books.xml-4            81.0MB/s ± 0%
XML/sample_catalog.xml-4          58.6MB/s ± 0%
XML/sample_omg.xml-4               159MB/s ± 0%

HTML

HTML (with JS and CSS) minification typically shaves off about 10%.

The HTML5 minifier uses these minifications:

strip unnecessary whitespace and otherwise collapse it to one space (or newline if it originally contained a newline)
strip superfluous quotes, or uses single/double quotes whichever requires fewer escapes
strip default attribute values and attribute boolean values
strip some empty attributes
strip unrequired tags (html, head, body, ...)
strip unrequired end tags (tr, td, li, ... and often p)
strip default protocols (http:, https: and javascript:)
strip all comments (including conditional comments, old IE versions are not supported anymore by Microsoft)
shorten doctype and meta charset
lowercase tags, attributes and some values to enhance gzip compression

Options:

KeepConditionalComments preserve all IE conditional comments such as  and <![if IE 6]><![endif]>, see https://msdn.microsoft.com/en-us/library/ms537512(v=vs.85).aspx#syntax
KeepDefaultAttrVals preserve default attribute values such as <script type="application/javascript">
KeepDocumentTags preserve html, head and body tags
KeepEndTags preserve all end tags
KeepQuotes preserve quotes around attribute values
KeepWhitespace preserve whitespace between inline tags but still collapse multiple whitespace characters into one

After recent benchmarking and profiling it became really fast and minifies pages in the 10ms range, making it viable for on-the-fly minification.

However, be careful when doing on-the-fly minification. Minification typically trims off 10% and does this at worst around about 20MB/s. This means users have to download slower than 2MB/s to make on-the-fly minification worthwhile. This may or may not apply in your situation. Rather use caching!

Whitespace removal

The whitespace removal mechanism collapses all sequences of whitespace (spaces, newlines, tabs) to a single space. If the sequence contained a newline or carriage return it will collapse into a newline character instead. It trims all text parts (in between tags) depending on whether it was preceded by a space from a previous piece of text and whether it is followed up by a block element or an inline element. In the former case we can omit spaces while for inline elements whitespace has significance.

Make sure your HTML doesn't depend on whitespace between block elements that have been changed to inline or inline-block elements using CSS. Your layout should not depend on those whitespaces as the minifier will remove them. An example is a menu consisting of multiple <li> that have display:inline-block applied and have whitespace in between them. It is bad practise to rely on whitespace for element positioning anyways!

CSS

Minification typically shaves off about 10%-15%. This CSS minifier will not do structural changes to your stylesheets. Although this could result in smaller files, the complexity is quite high and the risk of breaking website is high too.

The CSS minifier will only use safe minifications:

remove comments and unnecessary whitespace (but keep /*! ... */ which usually contains the license)
remove trailing semicolons
optimize margin, padding and border-width number of sides
shorten numbers by removing unnecessary + and zeros and rewriting with/without exponent
remove dimension and percentage for zero values
remove quotes for URLs
remove quotes for font families and make lowercase
rewrite hex colors to/from color names, or to three digit hex
rewrite rgb(, rgba(, hsl( and hsla( colors to hex or name
use four digit hex for alpha values (transparent → #0000)
replace normal and bold by numbers for font-weight and font
replace none → 0 for border, background and outline
lowercase all identifiers except classes, IDs and URLs to enhance gzip compression
shorten MS alpha function
rewrite data URIs with base64 or ASCII whichever is shorter
calls minifier for data URI mediatypes, thus you can compress embedded SVG files if you have that minifier attached
shorten aggregate declarations such as background and font

It does purposely not use the following techniques:

(partially) merge rulesets
(partially) split rulesets
collapse multiple declarations when main declaration is defined within a ruleset (don't put font-weight within an already existing font, too complex)
remove overwritten properties in ruleset (this not always overwrites it, for example with !important)
rewrite properties into one ruleset if possible (like margin-top, margin-right, margin-bottom and margin-left → margin)
put nested ID selector at the front (body > div#elem p → #elem p)
rewrite attribute selectors for IDs and classes (div[id=a] → div#a)
put space after pseudo-selectors (IE6 is old, move on!)

There are a couple of comparison tables online, such as CSS Minifier Comparison, CSS minifiers comparison and CleanCSS tests. Comparing speed between each, this minifier will usually be between 10x-300x faster than existing implementations, and even rank among the top for minification ratios. It falls short with the purposely not implemented and often unsafe techniques.

Options:

KeepCSS2 prohibits using CSS3 syntax (such as exponents in numbers, or rgba( → rgb(), might be incomplete
Precision number of significant digits to preserve for numbers, 0 means no trimming

JS

The JS minifier typically shaves off about 35% -- 65% of filesize depening on the file, which is a compression close to many other minifiers. Common speeds of PHP and JS implementations are about 100-300kB/s (see Uglify2, Adventures in PHP web asset minimization). This implementation is orders of magnitude faster at around ~25MB/s.

The following features are implemented:

remove superfluous whitespace
remove superfluous semicolons
shorten true, false, and undefined to !0, !1 and void 0
rename variables and functions to shorter names (not in global scope)
move var declarations to the top of the global/function scope (if more than one)
collapse if/else statements to expressions
minify conditional expressions to simpler ones
merge sequential expression statements to one, including into return and throw
remove superfluous grouping in expressions
shorten or remove string escapes
convert object key or index expression from string to identifier or decimal
merge concatenated strings
rewrite numbers (binary, octal, decimal, hexadecimal) to shorter representations

Options:

KeepVarNames keeps variable names as they are and omits shortening variable names
Precision number of significant digits to preserve for numbers, 0 means no trimming
Version ECMAScript version to use for output, 0 is the latest

Comparison with other tools

Performance is measured with time [command] ran 10 times and selecting the fastest one, on a Thinkpad T460 (i5-6300U quad-core 2.4GHz running Arch Linux) using Go 1.15.

minify: minify -o script.min.js script.js
esbuild: esbuild --minify --outfile=script.min.js script.js
terser: terser script.js --compress --mangle -o script.min.js
UglifyJS: uglifyjs --compress --mangle -o script.min.js script.js
Closure Compiler: closure-compiler -O SIMPLE --js script.js --js_output_file script.min.js --language_in ECMASCRIPT_NEXT -W QUIET --jscomp_off=checkVars optimization level SIMPLE instead of ADVANCED to make similar assumptions as do the other tools (do not rename/assume anything of global level variables)

Compression ratio (lower is better)

All tools give very similar results, although UglifyJS compresses slightly better.

Tool	ace.js	dot.js	jquery.js	jqueryui.js	moment.js
minify	53.7%	64.8%	34.2%	51.3%	34.8%
esbuild	53.8%	66.3%	34.4%	53.1%	34.8%
terser	53.2%	65.2%	34.2%	51.8%	34.7%
UglifyJS	53.1%	64.7%	33.8%	50.7%	34.2%
Closure Compiler	53.4%	64.0%	35.7%	53.6%	34.3%

Time (lower is better)

Most tools are extremely slow, with minify and esbuild being orders of magnitudes faster.

Tool	ace.js	dot.js	jquery.js	jqueryui.js	moment.js
minify	49ms	5ms	22ms	35ms	13ms
esbuild	64ms	9ms	31ms	51ms	17ms
terser	2900s	180ms	1400ms	2200ms	730ms
UglifyJS	3900ms	210ms	2000ms	3100ms	910ms
Closure Compiler	6100ms	2500ms	4400ms	5300ms	3500ms

JSON

Minification typically shaves off about 15% of filesize for common indented JSON such as generated by JSON Generator.

The JSON minifier only removes whitespace, which is the only thing that can be left out, and minifies numbers (1000 => 1e3).

Options:

Precision number of significant digits to preserve for numbers, 0 means no trimming
KeepNumbers do not minify numbers if set to true, by default numbers will be minified

SVG

The SVG minifier uses these minifications:

trim and collapse whitespace between all tags
strip comments, empty doctype, XML prelude, metadata
strip SVG version
strip CDATA sections wherever possible
collapse tags with no content to a void tag
minify style tag and attributes with the CSS minifier
minify colors
shorten lengths and numbers and remove default px unit
shorten path data
use relative or absolute positions in path data whichever is shorter

TODO:

convert attributes to style attribute whenever shorter
merge path data? (same style and no intersection -- the latter is difficult)

Options:

Precision number of significant digits to preserve for numbers, 0 means no trimming

XML

The XML minifier uses these minifications:

strip unnecessary whitespace and otherwise collapse it to one space (or newline if it originally contained a newline)
strip comments
collapse tags with no content to a void tag
strip CDATA sections wherever possible

Options:

KeepWhitespace preserve whitespace between inline tags but still collapse multiple whitespace characters into one

Usage

Any input stream is being buffered by the minification functions. This is how the underlying buffer package inherently works to ensure high performance. The output stream however is not buffered. It is wise to preallocate a buffer as big as the input to which the output is written, or otherwise use bufio to buffer to a streaming writer.

New

Retrieve a minifier struct which holds a map of mediatype → minifier functions.

m := minify.New()

The following loads all provided minifiers.

m := minify.New()
m.AddFunc("text/css", css.Minify)
m.AddFunc("text/html", html.Minify)
m.AddFunc("image/svg+xml", svg.Minify)
m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

You can set options to several minifiers.

m.Add("text/html", &html.Minifier{
	KeepDefaultAttrVals: true,
	KeepWhitespace: true,
})

From reader

Minify from an io.Reader to an io.Writer for a specific mediatype.

if err := m.Minify(mediatype, w, r); err != nil {
	panic(err)
}

From bytes

Minify from and to a []byte for a specific mediatype.

b, err = m.Bytes(mediatype, b)
if err != nil {
	panic(err)
}

From string

Minify from and to a string for a specific mediatype.

s, err = m.String(mediatype, s)
if err != nil {
	panic(err)
}

To reader

Get a minifying reader for a specific mediatype.

mr := m.Reader(mediatype, r)
if _, err := mr.Read(b); err != nil {
	panic(err)
}

To writer

Get a minifying writer for a specific mediatype. Must be explicitly closed because it uses an io.Pipe underneath.

mw := m.Writer(mediatype, w)
if mw.Write([]byte("input")); err != nil {
	panic(err)
}
if err := mw.Close(); err != nil {
	panic(err)
}

Middleware

Minify resources on the fly using middleware. It passes a wrapped response writer to the handler that removes the Content-Length header. The minifier is chosen based on the Content-Type header or, if the header is empty, by the request URI file extension. This is on-the-fly processing, you should preferably cache the results though!

fs := http.FileServer(http.Dir("www/"))
http.Handle("/", m.Middleware(fs))

Custom minifier

Add a minifier for a specific mimetype.

type CustomMinifier struct {
	KeepLineBreaks bool
}

func (c *CustomMinifier) Minify(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
}

m.Add(mimetype, &CustomMinifier{KeepLineBreaks: true})
// or
m.AddRegexp(regexp.MustCompile("/x-custom$"), &CustomMinifier{KeepLineBreaks: true})

Add a minify function for a specific mimetype.

m.AddFunc(mimetype, func(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
})
m.AddFuncRegexp(regexp.MustCompile("/x-custom$"), func(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
})

Add a command cmd with arguments args for a specific mimetype.

m.AddCmd(mimetype, exec.Command(cmd, args...))
m.AddCmdRegexp(regexp.MustCompile("/x-custom$"), exec.Command(cmd, args...))

Mediatypes

Using the params map[string]string argument one can pass parameters to the minifier such as seen in mediatypes (type/subtype; key1=val2; key2=val2). Examples are the encoding or charset of the data. Calling Minify will split the mimetype and parameters for the minifiers for you, but MinifyMimetype can be used if you already have them split up.

Minifiers can also be added using a regular expression. For example a minifier with image/.* will match any image mime.

Examples

Common minifiers

Basic example that minifies from stdin to stdout and loads the default HTML, CSS and JS minifiers. Optionally, one can enable java -jar build/compiler.jar to run for JS (for example the ClosureCompiler). Note that reading the file into a buffer first and writing to a pre-allocated buffer would be faster (but would disable streaming).

package main

import (
	"log"
	"os"
	"os/exec"

	"github.com/tdewolff/minify/v2"
	"github.com/tdewolff/minify/v2/css"
	"github.com/tdewolff/minify/v2/html"
	"github.com/tdewolff/minify/v2/js"
	"github.com/tdewolff/minify/v2/json"
	"github.com/tdewolff/minify/v2/svg"
	"github.com/tdewolff/minify/v2/xml"
)

func main() {
	m := minify.New()
	m.AddFunc("text/css", css.Minify)
	m.AddFunc("text/html", html.Minify)
	m.AddFunc("image/svg+xml", svg.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

	if err := m.Minify("text/html", os.Stdout, os.Stdin); err != nil {
		panic(err)
	}
}

External minifiers

Below are some examples of using common external minifiers.

Closure Compiler

See Closure Compiler Application. Not tested.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("java", "-jar", "build/compiler.jar"))

UglifyJS

See UglifyJS.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("uglifyjs"))

esbuild

See esbuild.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("esbuild", "$in.js", "--minify", "--outfile=$out.js"))

Custom minifier

Custom minifier showing an example that implements the minifier function interface. Within a custom minifier, it is possible to call any minifier function (through m minify.Minifier) recursively when dealing with embedded resources.

package main

import (
	"bufio"
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/tdewolff/minify/v2"
)

func main() {
	m := minify.New()
	m.AddFunc("text/plain", func(m *minify.M, w io.Writer, r io.Reader, _ map[string]string) error {
		// remove newlines and spaces
		rb := bufio.NewReader(r)
		for {
			line, err := rb.ReadString('\n')
			if err != nil && err != io.EOF {
				return err
			}
			if _, errws := io.WriteString(w, strings.Replace(line, " ", "", -1)); errws != nil {
				return errws
			}
			if err == io.EOF {
				break
			}
		}
		return nil
	})

	in := "Because my coffee was too cold, I heated it in the microwave."
	out, err := m.String("text/plain", in)
	if err != nil {
		panic(err)
	}
	fmt.Println(out)
	// Output: Becausemycoffeewastoocold,Iheateditinthemicrowave.
}

ResponseWriter

Middleware

func main() {
	m := minify.New()
	m.AddFunc("text/css", css.Minify)
	m.AddFunc("text/html", html.Minify)
	m.AddFunc("image/svg+xml", svg.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

	fs := http.FileServer(http.Dir("www/"))
	http.Handle("/", m.MiddlewareWithError(fs))
}

func handleError(w http.ResponseWriter, r *http.Request, err error) {
    http.Error(w, err.Error(), http.StatusInternalServerError)
}

In order to properly handle minify errors, it is necessary to close the response writer since all writes are concurrently handled. There is no need to check errors on writes since they will be returned on closing.

func main() {
	m := minify.New()
	m.AddFunc("text/html", html.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)

	input := `<script>const i = 1_000_</script>` // Faulty JS
	req := httptest.NewRequest(http.MethodGet, "/", nil)
	rec := httptest.NewRecorder()
	m.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/html")
		_, _ = w.Write([]byte(input))

		if err = w.(io.Closer).Close(); err != nil {
			panic(err)
		}
	})).ServeHTTP(rec, req)
}

ResponseWriter

func Serve(w http.ResponseWriter, r *http.Request) {
	mw := m.ResponseWriter(w, r)
	defer mw.Close()
	w = mw

	http.ServeFile(w, r, path.Join("www", r.URL.Path))
}

Custom response writer

ResponseWriter example which returns a ResponseWriter that minifies the content and then writes to the original ResponseWriter. Any write after applying this filter will be minified.

type MinifyResponseWriter struct {
	http.ResponseWriter
	io.WriteCloser
}

func (m MinifyResponseWriter) Write(b []byte) (int, error) {
	return m.WriteCloser.Write(b)
}

// MinifyResponseWriter must be closed explicitly by calling site.
func MinifyFilter(mediatype string, res http.ResponseWriter) MinifyResponseWriter {
	m := minify.New()
	// add minfiers

	mw := m.Writer(mediatype, res)
	return MinifyResponseWriter{res, mw}
}

// Usage
func(w http.ResponseWriter, req *http.Request) {
	w = MinifyFilter("text/html", w)
	if _, err := io.WriteString(w, "<p class="message"> This HTTP response will be minified. </p>"); err != nil {
		panic(err)
	}
	if err := w.Close(); err != nil {
		panic(err)
	}
	// Output: <p class=message>This HTTP response will be minified.
}

Templates

Here's an example of a replacement for template.ParseFiles from template/html, which automatically minifies each template before parsing it.

Be aware that minifying templates will work in most cases but not all. Because the HTML minifier only works for valid HTML5, your template must be valid HTML5 of itself. Template tags are parsed as regular text by the minifier.

func compileTemplates(filenames ...string) (*template.Template, error) {
	m := minify.New()
	m.AddFunc("text/html", html.Minify)

	var tmpl *template.Template
	for _, filename := range filenames {
		name := filepath.Base(filename)
		if tmpl == nil {
			tmpl = template.New(name)
		} else {
			tmpl = tmpl.New(name)
		}

		b, err := ioutil.ReadFile(filename)
		if err != nil {
			return nil, err
		}

		mb, err := m.Bytes("text/html", b)
		if err != nil {
			return nil, err
		}
		tmpl.Parse(string(mb))
	}
	return tmpl, nil
}

Example usage:

templates := template.Must(compileTemplates("view.html", "home.html"))

FAQ

Newlines remain in minified output

While you might expect the minified output to be on a single line for it to be fully minified, this is not true. In many cases, using a literal newline doesn't affect the file size, and in some cases it may even reduce the file size.

A typical example is HTML. Whitespace is significant in HTML, meaning that spaces and newlines between or around tags may affect how they are displayed. There is no distinction between a space or a newline and they may be interchanged without affecting the displayed HTML. Remember that a space (0x20) and a newline (0x0A) are both one byte long, so that there is no difference in file size when interchanging them. This minifier removes unnecessary whitespace by replacing stretches of spaces and newlines by a single whitespace character. Specifically, if the stretch of white space characters contains a newline, it will replace it by a newline and otherwise by a space. This doesn't affect the file size, but may help somewhat for debugging or file transmission objectives.

Another example is JavaScript. Single or double quoted string literals may not contain newline characters but instead need to escape them as \n. These are two bytes instead of a single newline byte. Using template literals it is allowed to have literal newline characters and we can use that fact to shave-off one byte! The result is that the minified output contains newlines instead of escaped newline characters, which makes the final file size smaller. Of course, changing from single or double quotes to template literals depends on other factors as well, and this minifier makes a calculation whether the template literal results in a shorter file size or not before converting a string literal.

License

Released under the MIT license.

README.md

Minify

Sponsors

Table of Contents

Roadmap

Prologue

Installation

Docker

API stability

Testing

Performance

HTML

Whitespace removal

CSS

JS

Comparison with other tools

Compression ratio (lower is better)

Time (lower is better)

JSON

SVG

XML

Usage

New

From reader

From bytes

From string

To reader

To writer

Middleware

Custom minifier

Mediatypes

Examples

Common minifiers

External minifiers

Closure Compiler

UglifyJS

esbuild

Custom minifier

ResponseWriter

Middleware

ResponseWriter

Custom response writer

Templates

FAQ

Newlines remain in minified output

License