diff --git a/README.md b/README.md index 96233f30..2e1ca9c1 100644 --- a/README.md +++ b/README.md @@ -3,92 +3,92 @@ Readium LCP Server Documentation ============ -As a retailer, public library or specialized e-distributor, you are distributing EPUB and PDF files and you want them protected against oversharing by the Readium LCP DRM. Your distribution platform already handles publications, users and the purchases / loans and you can develop a REST interface between this platform and a License server. If you are in this situation, the open-source codebase is what you need. +As a retailer, public library or specialized e-distributor, you are distributing EPUB or PDF ebooks, LPF or RPF packaged audiobooks or comics. You want them protected by the Readium LCP DRM. Your CMS (Content Management system) already handles publications, users, purchases or loans, your technical team is able to integrate this CMS with a License server by creating a new endpoint in the CMS and requesting the License Server via its REST interface. If you are in this situation, this open-source codebase is made for you. Using the Readium LCP Server you can: -* Encrypt your entire catalog of EPUB and PDF files and make these files ready for download from any LCP compliant user agents; -* Generate LCP licenses on the fly from your distribution platform, which will deliver them to the proper users and user agents; -* Let users request a loan extension or make an early loan return; -* Cancel a license in case a user has declared he wasn't able to user it (usually because he does not use an LCP compliant reading application); -* Revoke a license in case a user has overshared it. +* Encrypt your entire catalog of publications and store these encrypted files in a file system or S3 bucket, ready for download from any LCP compliant reading application; +* Generate LCP licenses and get up-to-date licenses; +* Let users request a loan extension or an early return; +* Cancel a license in case a user has declared he wasn't able to use it; +* Revoke a license in case of oversharing. -Detailed documentation can be found in the [Wiki pages](../../wiki) of the project. +**A detailed documentation is found in the [Wiki pages of the project](../../wiki). You really have to read it before you start testing this application.** Prerequisites ============= -No binaries are currently pre-built, so you need to get a working Golang installation. -Please refer to the official documentation for installation procedures at https://golang.org/. +Binaries are not yet pre-built, so you need to get a working Golang installation. +Please refer to the official GO documentation for installation procedures at https://golang.org/. -Install *go 1.13* or higher. +This software is working with *go 1.13* or higher. The servers require the setup of an SQL Database. - SQLite is sufficient for most needs. If the "database" property of each server defines a sqlite3 driver, the db setup is dynamically achieved when the server runs for the first time. SQLite database creation scripts are provided in the "dbmodel" folder in case they are useful. -- MySQL database creation scripts are provided as well in the "dbmodel" folder. These scripts should be run before launching the servers for the first time. +- MySQL database creation scripts are provided as well in the "dbmodel" folder. These scripts should be applied before launching the servers for the first time. -We expect other drivers (PostgresQL ...) to be provided by the community. Some developers have deployed MS SQL Server, but the corresponding scripts were not provided so far. - -A major revision of the software will feature an ORM (Object Realtional Mapper), but it is still unsufficiently tested to be moved to the master branch. +We expect other drivers (PostgresQL ...) to be provided by the community. Some implementers have deployed MS SQL Server, but the corresponding scripts were not provided so far. More db connectors will be provided in a future major revision of the application. Your platform must be able to handle: -1/ the license server, active in your intranet, not accessible from the Web, only accessible from you frontend server via its REST API. +1/ the License Server, active in your intranet, not accessible from the Web, only accessible from your CMS via a REST API. -2/ the license status server, accessible from the Web via its REST API. +2/ the License Status Server, accessible from the Web via a REST API. -3/ a large storage volume for encrypted publications, accessible in read mode from the Web via HTTP URLs (publications are encrypted once, every license generated for such publication is pointing at the same encrypted file. +3/ a large storage volume for encrypted publications (file system or S3 bucket), accessible from the Web via HTTP URLs. Note that publications are encrypted once: every license generated for such publication is pointing at the same encrypted file. Because these publications are stronlgy encrypted and the decryption key is secured in your SQL database, public access to these files is not problematic. -You must obtain a X.509 certificate and confidential crypto information through EDRLab in order for your licenses to be accepted by Readium LCP compliant Reading Systems. +Encryption Profiles +=================== +Out of the box, this open-source software is using what we call the "basic" LCP profile, i.e. a testing mode provided by the [LCP open standard](https://readium.org/lcp-specs/). Licenses generated with this "basic" profile are perfectly handled by reading applications based on [Readium Mobile](https://www.edrlab.org/software/readium-mobile/), as well as by [Thorium Reader](https://www.edrlab.org/software/thorium-reader/). -A folder publicly accessible from the Web must be made available for the server to store encrypted files. +But this profile, because it is open, does not offer any security. Security is provided by a "production" profile, i.e. confidential crypto information and a personal X.509 certificate delivered to trusted implementers by [EDRLab](mailto:contact@edrlab.org), the wordwide LCP Certificcation Authority. Licenses generated with the "production" profile are handled by any LCP compliant Reading System. Executables =========== -The server software is composed of three independant parts: +The server software is composed of several independant parts: ## [lcpencrypt] A command line utility for content encryption. This utility can be included in any processing pipeline. -lcpencrypt: -* Takes an unprotected publication as input and generates an encrypted file as output. -* Notifies the License server of the generation of the encrypted file. +lcpencrypt can: +* Take an unprotected publication as input and generates an encrypted file as output +* Optionally, store the encrypted file into a file system or S3 bucket +* Notifie the License server of the generation of the encrypted file ## [lcpserver] -A License server, which implements Readium Licensed Content Protection 1.0. +A License server implements [Readium Licensed Content Protection 1.0](https://readium.org/lcp-specs/releases/lcp/latest). -Private functionalities (authentication needed): -* Store the data resulting from an external encryption -* Generate a license -* Generate a protected publication +Private functionalities (authentication required) are: +* Store the data resulting from an external encryption, if the encryption utility did not already store it +* Generate a license or returns an up-to-date license +* Generate a protected publication (i.e. an encrypted publication in which a license is embedded) * Update the rights associated with a license * Get a set of licenses * Get a license ## [lsdserver] -A License Status server, which implements Readium License Status Document 1.0. +A License Status server implements [Readium License Status Document 1.0](https://readium.org/lcp-specs/releases/lsd/latest). -Public functionalities (accessible from the web): +Public functionalities (accessible from the web) are: * Return a license status document -* Process a device registration -* Process a lending return -* Process a lending renewal +* Process a device registration request +* Process a lending return request +* Process a lending renewal request -Private functionalities (authentication needed): -* Create a license status document -* Filter licenses +Private functionalities (authentication required) are: +* Be notified of the generation of a new license +* Filter licenses by count of registered devices * List all registered devices for a given licence -* Revoke/cancel a license - +* Revoke or cancel a license ## [frontend] -A Test Frontend server, which mimics your own frontend platform (e.g. bookselling website), with a GUI and its own REST API. Its sole goal is to help you test the License and License status servers. +A Test Frontend server is also provided, which mimics your own CMS, with a GUI and its own REST API. Its sole goal is to help you test the License and License Status servers. It should never be used in production. -Public functionalities (accessible from the web): +Public functionalities (accessible from the web) are: * Fetch a license from its id * Fetch a licensed publication from the license id @@ -96,7 +96,7 @@ Public functionalities (accessible from the web): Install ======= -Assuming a working Go installation and a properly set $GOPATH, the following will install the four executables that constitute a complete Readium LCP Server. +Assuming a working Go installation ... On Linux and MacOS: @@ -190,7 +190,7 @@ A test certificate (`cert-edrlab-test.pem`) and private key (`privkey-edrlab-tes A quick-start configuration meant only for test purposes is available in `test/config.yaml`. This file includes a default configuration for the the LCP, LSD and frontend servers. -1. Create a LCP_HOME folder, eg. `/usr/local/var/lcp` +1. Create a `` folder, eg. `/usr/local/var/lcp` 2. Copy the `test/config.yaml` file into LCP_HOME 3. Replace any occurrence of `` in config.yaml with the absolute path to the LCP_HOME folder 4. Setup the `READIUM_*_CONFIG` env variables, which must reference the configuration file @@ -204,86 +204,90 @@ Here are the details about the configuration properties of each server. In the s ### License Server +#### profile section `profile`: value of the LCP profile; allowed values are: -- `basic`: default value, as described in the Readium LCP specification, used for tests only -- `1.0`: the current production profile, created by EDRLab. - -`lcp` section: parameters associated with the License Server. -- `host`: the public server hostname, `hostname` by default -- `port`: the listening port, `8989` by default -- `public_base_url`: the public base URL, used by the license status server and the frontend test server to communicate with this server; combination of the host and port values on http by default, which is sufficient as the license server should not be visible from the Web. -- `database`: the URI formatted connection string to the database, `sqlite3://file:lcp.sqlite?cache=shared&mode=rwc` by default +- `basic`: default value, as described in the Readium LCP specification, used for tests only. +- `1.0`: the current production profile, maintained by EDRLab. + +#### lcp section +`lcp`: parameters associated with the License Server. +- `host`: the public server hostname, `hostname` by default. +- `port`: the listening port, `8989` by default. +- `public_base_url`: the URL used by the License Status Server and the Frontend Test Server to communicate with this License server; combination of the host and port values on http by default. +- `database`: the URI formatted connection string to the database, `sqlite3://file:lcp.sqlite?cache=shared&mode=rwc` by default. `mysql://login:password@/dbname?parseTime=true` if your using MySQL. - `auth_file`: mandatory; the path to the password file introduced above. -`storage` section: parameters related to the storage of encrypted publications. -- `mode` : optional. Possible values are "local" (default value) and "s3". +#### storage section +This section should be empty if the storage location of encrypted publications is managed by the lcpencrypt utility. +If this section is present and lcpencrypt does not manage the storage, all encrypted publications will be stored in the configured folder or s3 bucket. -If `mode` value is `s3`: -- `endpoint` (optional): name of the target S3 endpoint, if one is defined in the AWS S3 setup. -- `region` (required): name of the target AWS region. -- `bucket` (required): name of the target S3 bucket. +`storage`: parameters related to the storage of encrypted publications. +- `mode` : optional. Possible values are "fs" (default value) and "s3". -If the storage is an S3 bucket, client credentials default to a chain of credential providers, searched in environment variables and a shared credential file. See [Setting up an S3 Storage](https://github.com/readium/readium-lcp-server/wiki/Setting-up-an-S3-storage) for details. +If `mode` value is `s3`, the following parameters are expected: +- `bucket` (required): name of the target S3 bucket. +- `region` (optional): name of the target AWS region. -Alternatively (and this is not recommended!), credentials can be stored in clear in the configuration file: +The S3 region and client credentials default to a chain of credential providers, searched in environment variables and shared files. See [Setting up an S3 Storage](https://github.com/readium/readium-lcp-server/wiki/Setting-up-an-S3-storage) for details. +Alternatively (but this is not recommended!), credentials can be stored in clear in the configuration file: - `access_id`: value of the AWS access key id. - `secret`: value of the AWS secret access key. -If `mode` value is NOT `s3`: +If `mode` value is NOT `s3`, the following paremeters are expected: - `filesystem` subsection: parameters related to a file system storage. - - `directory`: absolute path to the directory in which the encrypted publications are stored. In production, this directory must be accessible from the Web via the URL defined in `license/links/publication` (see below) - This storage must be accessible from the Web via a simple URL, specified via the `license/publication` parameter. + - `directory`: absolute path of the directory in which all encrypted publications are stored. + - `url`: absolute http or https url of the storage volume in which all encrypted publications are stored. -`certificate` section: parameters related to the signature of licenses: +#### certificate section +`certificate`: parameters related to the signature of licenses: - `cert`: the path to provider certificate file (.pem or .crt). It will be inserted in the licenses and used by clients for checking the signature. - `private_key`: the path to the private key (.pem) asociated with the certificate. It will be used for signing licenses. -`license` section: parameters related to static information to be included in all licenses generated by the License Server: +#### license section +`license`: parameters related to static information to be included in all licenses generated by the License Server: - `links`: subsection: links that will be included in all licenses. `hint` and `publication` links are required in a Readium LCP license. If no such link exists in the partial license passed from the frontend when a new license his requested, these link values will be inserted in the partial license. If no value is present in the configuration file and no value is inserted in the partial license, the License server will reply with a 500 Server Error at license creation. The sub-properties of the `links` section are: - - `hint`: required; location where a Reading System can redirect a User looking for additional information about the User Passphrase. - - `publication`: optional, templated URL; - location where the encrypted Publication associated with the License Document will be downloaded from the Web. - This access point corresponds to the directory where encrypted publications are stored by the License Server (see `storage/filesystem/directory`). - To expose this storage directory on the Web, the provider may decide to install a reverse-proxy, use a Web drive, use a CDN etc. This is a deployment choice which has nothing to do with this open-source projet. - During initial tests (before the License Server is hidden from the Web), this URL may simply be the one described described [here](https://github.com/readium/readium-lcp-server/wiki/LCP-License-Server-API#fetch-an-encrypted-publication). - The publication (alias content) identifier is inserted in the URL via the variable {publication_id}. - Note that this is working because the file name of the stored encrypted publications is simply their publication identifier. - - `status`: optional, templated URL; location of the Status Document associated with a License Document. - The license identifier is inserted via the variable {license_id}. - -`lsd_notify_auth` section: authentication parameters used by the License Server for notifying the License Status Server -of a license generation. The notification endpoint is configured in the `lsd` section. -- `username`: mandatory, authentication username -- `password`: mandatory, authentication password - -Here is a License Server sample config (assuming the License Status Server is using the 'basic' LCP profile, is active on http://127.0.0.1:8990 and the Frontend Server is active on http://127.0.0.1:8991): -```json + - `status`: required, templated URL; location of the Status Document associated with a License Document. + The license identifier is inserted via the `{license_id}` variable. + - `hint`: required; location where a Reading System can redirect a user looking for additional information about the User Passphrase. + - `publication`: *deprecated in favor of the storage / filesystem / url parameter*, templated URL; + Absolute http or https url of the storage volume in which all encrypted publications are stored. + The publication identifier is inserted via the `{publication_id}` variable. + +#### lsd and lsd_notify_auth section +`lsd_notify_auth`: authentication parameters used by the License Server for notifying the License Status Server +of the generation of a new license. The notification endpoint is configured in the `lsd` section. +- `username`: required, authentication username +- `password`: required, authentication password + +#### Sample config +Here is a License Server sample config: + +```yaml profile: "basic" lcp: - host: "127.0.0.1" + host: "192.168.0.1" port: 8989 - public_base_url: "http://127.0.0.1:8989" - database: "sqlite3://file:/db/lcp.sqlite?cache=shared&mode=rwc" - auth_file: "/htpasswd" + public_base_url: "http://192.168.0.1:8989/lcpserver" + database: "sqlite3://file:/usr/local/var/lcp/db/lcp.sqlite?cache=shared&mode=rwc" + auth_file: "/usr/local/var/lcp/lcpsv/htpasswd" storage: filesystem: - directory: "/files/storage" + directory: "/usr/local/var/lcp/storage" + url: "https://www.example.net/lcp/files/storage/" certificate: - cert: "/cert/cert.pem" - private_key: "/cert/privkey.pem" + cert: "/usr/local/var/lcp/cert/cert.pem" + private_key: "/usr/local/var/lcp/cert/privkey.pem" license: links: - status: "http://127.0.0.1:8990/licenses/{license_id}/status" - hint: "http://127.0.0.1:8991/static/hint.html" - publication: "http://127.0.0.1:8989/contents/{publication_id}" - + status: "https://www.example.net/lsdserver/licenses/{license_id}/status" + hint: "https://www.example.net/static/lcp_hint.html" lsd: - public_base_url: "http://127.0.0.1:8990" + public_base_url: "http://192.168.0.1:8990" lsd_notify_auth: username: "adm_username" password: "adm_password" @@ -292,16 +296,18 @@ lsd_notify_auth: ### License Status Server -`lsd` section: parameters associated with the License Status Server. -- `host`: the public server hostname, `hostname` by default -- `port`: the listening port, `8990` by default -- `public_base_url`: the public base URL, used by the license server and the frontend test server to communicate with this server; combination of the host and port values on http by default; as this server is exposed on the Web in production, a domain name should be present in the URL. -- `database`: the URI formatted connection string to the database, `sqlite3://file:lsd.sqlite?cache=shared&mode=rwc` by default +#### lsd section +`lsd`: parameters associated with the License Status Server. +- `host`: the public server hostname, `hostname` by default. +- `port`: the listening port, `8990` by default. +- `public_base_url`: the URL used by the License Server and the Frontend Test Server to communicate with this License Status Server; combination of the host and port values on http by default. +- `database`: the URI formatted connection string to the database, `sqlite3://file:lsd.sqlite?cache=shared&mode=rwc` by default. `mysql://login:password@/dbname?parseTime=true` if your using MySQL. - `auth_file`: mandatory; the path to the password file introduced above. - `license_link_url`: mandatory; the url template representing the url from which a license can be fetched from the provider's frontend server. This url will be inserted in the 'license' link of every status document. It must be the url of a server acting as a proxy between the user request and the License Server. Such proxy is mandatory, as the License Server does not possess user information needed to craft a license from its identifier. If the test frontend server is used as a proxy, the url must be of the form "http:///api/v1/licenses/{license_id}" (note the /api/v1 section). -`license_status` section: parameters related to the interactions implemented by the License Status server, if any: +#### license_status section +`license_status`: parameters related to the interactions implemented by the License Status server, if any: - `renting_days`: maximum number of days allowed for a loan, from the date the loan starts. If set to 0 or absent, no loan renewal is possible. - `renew`: boolean; if `true`, the renewal of a loan is possible. - `renew_days`: default number of additional days allowed during a renewal. @@ -309,20 +315,22 @@ lsd_notify_auth: - `register`: boolean; if `true`, registering a device is possible. - `renew_page_url`: URL; if set, the renew feature is implemented as an HTML page, using this URL. This is mostly useful for testing client applications. -`lcp_update_auth` section: authentication parameters used by the License Status Server for updating a license via the License Server. The notification endpoint is configured in the `lcp` section. +#### lcp_update_auth section +`lcp_update_auth`: authentication parameters used by the License Status Server for updating a license via the License Server. The notification endpoint is configured in the `lcp` section. - `username`: mandatory, authentication username - `password`: mandatory, authentication password -Here is a License Status Server sample config (assuming the License Status Server is active on http://127.0.0.1:8990 and the Frontend Server is active on http://127.0.0.1:8991): +#### Sample config +Here is a License Status Server sample config: -```json +```yaml lsd: - host: "127.0.0.1" + host: "192.168.0.1" port: 8990 public_base_url: "http://127.0.0.1:8990" - database: "sqlite3://file:/db/lsd.sqlite?cache=shared&mode=rwc" - auth_file: "/htpasswd" - license_link_url: "http://127.0.0.1:8991/api/v1/licenses/{license_id}" + database: "sqlite3://file:/usr/local/var/lcp/db/lsd.sqlite?cache=shared&mode=rwc" + auth_file: "/usr/local/var/lcp/htpasswd" + license_link_url: "https://www.example.net/lcp/licenses/{license_id}" license_status: register: true renew: true @@ -339,11 +347,12 @@ lcp_update_auth: ### Frontend Server -`frontend` section: parameters associated with the Test Frontend Server. +#### frontend section +`frontend`: parameters associated with the Test Frontend Server. - `host`: the public server hostname, `hostname` by default - `port`: the listening port, `8991` by default -- `public_base_url`: the public base URL, used to access the frontend UI; combination of the host and port values on http by default -- `database`: the URI formatted connection string to the database, `sqlite3://file:frontend.sqlite?cache=shared&mode=rwc` by default +- `public_base_url`: the URL used by the Frontend node.js software to communicate with this Frontend Test Server; combination of the host and port values on http by default. +- `database`: the URI formatted connection string to the database, `sqlite3://file:frontend.sqlite?cache=shared&mode=rwc` by default. `mysql://login:password@/dbname?parseTime=true` if your using MySQL. - `master_repository`: repository where the uploaded EPUB files are stored before encryption. - `encrypted_repository`: repository where the encrypted EPUB files are stored after upload. The LCP server must have access to the path declared here, as it will move each encrypted file to its final storage folder on notification of encryption from the Frontend Server. - `directory`: the directory containing the client web app; by default $GOPATH/src/github.com/readium/readium-lcp-server/frontend/manage. @@ -353,6 +362,9 @@ lcp_update_auth: The config file of a Test Frontend Server must also define the following properties: +#### other required sections +The Test Frontend Server must communicate with the License Server and the License Status Server. This is why it must contain the following sections, with the values defined above. + `lcp` - `public_base_url` @@ -367,22 +379,24 @@ The config file of a Test Frontend Server must also define the following propert - `username` - `password` +#### Sample config Here is a Test Frontend Server sample config: -```json + +```yaml frontend: - host: "127.0.0.1" + host: "192.168.0.1" port: 8991 - database: "sqlite3://file:/db/frontend.sqlite?cache=shared&mode=rwc" - master_repository: "/files/master" - encrypted_repository: "/files/encrypted" - provider_uri: "https://www.myprovidername.org" + database: "sqlite3://file:/usr/local/var/lcp/db/frontend.sqlite?cache=shared&mode=rwc" + master_repository: "/usr/local/var/lcp/files/master" + encrypted_repository: "/usr/local/var/lcp/files/encrypted" + provider_uri: "https://www.example.net" right_print: 10 right_copy: 2000 lcp: - public_base_url: "http://127.0.0.1:8989" + public_base_url: "http://192.168.0.1:8989" lsd: - public_base_url: "http://127.0.0.1:8990" + public_base_url: "http://192.168.0.1:8990" lcp_update_auth: username: "adm_username" password: "adm_password" diff --git a/config/config.go b/config/config.go index 02483768..ac28451f 100644 --- a/config/config.go +++ b/config/config.go @@ -92,6 +92,7 @@ type Certificate struct { type FileSystem struct { Directory string `yaml:"directory"` + URL string `yaml:"url,omitempty"` } type Storage struct { diff --git a/dbmodel/sqlite_db_setup_lcpserver.sql b/dbmodel/sqlite_db_setup_lcpserver.sql index 507a3103..20aa88f7 100644 --- a/dbmodel/sqlite_db_setup_lcpserver.sql +++ b/dbmodel/sqlite_db_setup_lcpserver.sql @@ -2,8 +2,8 @@ CREATE TABLE content ( id varchar(255) PRIMARY KEY NOT NULL, encryption_key varchar(64) NOT NULL, location text NOT NULL, -  length bigint, -  sha256 varchar(64), + length bigint, + sha256 varchar(64), "type" varchar(255) NOT NULL DEFAULT 'application/epub+zip' ); diff --git a/encrypt/process_encrypt.go b/encrypt/process_encrypt.go new file mode 100644 index 00000000..2a8ea567 --- /dev/null +++ b/encrypt/process_encrypt.go @@ -0,0 +1,416 @@ +// Copyright 2021 Readium Foundation. All rights reserved. +// Use of this source code is governed by a BSD-style license +// that can be found in the LICENSE file exposed on Github (readium) in the project repository. + +package encrypt + +import ( + "archive/zip" + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + + "github.com/readium/readium-lcp-server/crypto" + "github.com/readium/readium-lcp-server/epub" + apilcp "github.com/readium/readium-lcp-server/lcpserver/api" + "github.com/readium/readium-lcp-server/pack" + uuid "github.com/satori/go.uuid" +) + +// ProcessPublication encrypts a publication +// inputPath must contain a processable file extension (EPUB, PDF, LPF or RPF) +func ProcessPublication(contentID, inputPath, tempRepo, outputRepo, storageRepo, storageURL string) (*apilcp.LcpPublication, error) { + + var pub apilcp.LcpPublication + + // if contentID is not set, generate a random UUID + if contentID == "" { + uid, err := uuid.NewV4() + if err != nil { + return nil, err + } + contentID = uid.String() + } + pub.ContentID = contentID + + // if the input file is stored on a remote server, fetch it and store it into a temp folder + tempPath, err := fetchInputFile(inputPath, tempRepo, contentID) + if err != nil { + return nil, err + } + deleteTemp := false + if tempPath != "" { + deleteTemp = true + inputPath = tempPath + } + + // select a storage mode + pub.StorageMode = apilcp.Storage_none + if storageRepo != "" { + // S3 storage + if strings.HasPrefix(storageRepo, "s3:") { + pub.StorageMode = apilcp.Storage_s3 + // fs storage (not http) + } else { + pub.StorageMode = apilcp.Storage_fs + // create the storage folder + os.MkdirAll(storageRepo, os.ModePerm) //ignore the error, the folder can already exist + // the encrypted file will be directly generated inside the storage path + outputRepo = storageRepo + } + } + + var outputPath string + // if the output repo is not set, the target file will be created + // inside the current working directory with the content id as file name. + if outputRepo == "" { + workingDir, _ := os.Getwd() + outputPath = filepath.Join(workingDir, pub.ContentID) + // replace any file name found in the output path by the content id + } else if filepath.Ext(outputRepo) != "" { + outputPath = filepath.Join(filepath.Dir(outputRepo), pub.ContentID) + // use the output repo as-is + } else { + outputPath = filepath.Join(outputRepo, pub.ContentID) + } + + // set target file info + targetFileInfo(&pub, inputPath) + + // define an AES encrypter + encrypter := crypto.NewAESEncrypter_PUBLICATION_RESOURCES() + + // select the encryption process from the input file extension + err = nil + inputExt := filepath.Ext(inputPath) + + switch inputExt { + case ".epub": + err = processEPUB(&pub, inputPath, outputPath, encrypter) + case ".pdf": + err = processPDF(&pub, inputPath, outputPath, encrypter) + case ".lpf": + err = processLPF(&pub, inputPath, outputPath, encrypter) + case ".audiobook", ".divina", ".rpf": + err = processRPF(&pub, inputPath, outputPath, encrypter) + } + if err != nil { + return nil, err + } + + if deleteTemp { + err = os.Remove(inputPath) + if err != nil { + return nil, err + } + } + + // store the publication if required, and set pub.Output + switch pub.StorageMode { + case apilcp.Storage_none: + // reminder: if the license server is requested storing the encrypted publication, + // then it must have read access to the output repo. + pub.Output = outputPath + case apilcp.Storage_fs: + // url of the publication + pub.Output, err = setPubURL(storageURL, pub.ContentID) + case apilcp.Storage_s3: + // store the encrypted file in its definitive S3 storage. + err = StorePublication(&pub, outputPath, storageRepo) + if err != nil { + return nil, err + } + // url of the publication + pub.Output, err = setPubURL(storageURL, pub.ContentID) + } + if err != nil { + return nil, err + } + return &pub, nil +} + +// fetchInputFile fetches the input file from a remote server +func fetchInputFile(inputPath, tempRepo, contentID string) (string, error) { + + url, err := url.Parse(inputPath) + if err != nil { + return "", err + } + + // no need to fetch the file, which is in a file system + if url.Scheme != "http" && url.Scheme != "https" && url.Scheme != "ftp" { + return "", nil + } + + // create a temp repo if needed + if tempRepo == "" { + tempRepo, _ = os.Getwd() + } + // the temp file has the same extension as the remote file + inputExt := filepath.Ext(inputPath) + tempPath := filepath.Join(tempRepo, contentID+inputExt) + // create the temp file + out, err := os.Create(tempPath) + if err != nil { + return "", err + } + defer out.Close() + + // fetch the file + if url.Scheme == "http" || url.Scheme == "https" { + res, err := http.Get(inputPath) + if err != nil { + return "", err + } + defer res.Body.Close() + _, err = io.Copy(out, res.Body) + if err != nil { + return "", err + } + } else if url.Scheme == "ftp" { + // we'll use https://github.com/jlaffaye/ftp when requested + return "", errors.New("ftp not supported yet") + } + return tempPath, nil +} + +// targetFileInfo set the content type and +// the file name which will be used during future downloads +// from the extension of the source file. +func targetFileInfo(pub *apilcp.LcpPublication, inputPath string) error { + + inputFile := filepath.Base(inputPath) + inputExt := filepath.Ext(inputPath) + fileNameNoExt := inputFile[:len(inputFile)-len(inputExt)] + + var ext, contentType string + switch inputExt { + case ".epub": + ext = inputExt + contentType = epub.ContentType_EPUB + case ".pdf": + ext = "lcpdf" + contentType = "application/pdf+lcp" + case ".audiobook": + ext = "lcpau" + contentType = "application/audiobook+lcp" + case ".divina": + ext = "lcpdi" + contentType = "application/divina+lcp" + case ".lpf": + // short term solution. We'll need to inspect the manifest and check conformsTo, + // to be certain this is an audiobook (vs another profile of Web Publication) + ext = "lcpau" + contentType = "application/audiobook+lcp" + case ".rpf": + // short term solution. We'll need to inspect the manifest and check conformsTo, + // to be certain this package contains a pdf + ext = "lcpdf" + contentType = "application/pdf+lcp" + } + pub.FileName = fileNameNoExt + ext + pub.ContentType = contentType + return nil +} + +// setPubURL sets a publication url from a base url and an id +func setPubURL(base, id string) (pubURL string, err error) { + + if base != "" { + base, err := url.Parse(base) + if err != nil { + return "", err + } + u, err := base.Parse(id) + if err != nil { + return "", err + } + pubURL = u.String() + } + return pubURL, nil +} + +// checksum calculates the checksum of a file +func checksum(file *os.File) string { + + hasher := sha256.New() + file.Seek(0, 0) + if _, err := io.Copy(hasher, file); err != nil { + return "" + } + return hex.EncodeToString(hasher.Sum(nil)) +} + +// processEPUB encrypts resources in an EPUB +func processEPUB(pub *apilcp.LcpPublication, inputPath string, outputPath string, encrypter crypto.Encrypter) error { + + // create a zip reader from the input path + zr, err := zip.OpenReader(inputPath) + if err != nil { + return err + } + defer zr.Close() + + // generate an EPUB object + epub, err := epub.Read(&zr.Reader) + if err != nil { + return err + } + // create the output file + outputFile, err := os.Create(outputPath) + if err != nil { + return err + } + // will close the output file + defer outputFile.Close() + // encrypt the content of the publication, + // write into the output file + _, encryptionKey, err := pack.Do(encrypter, epub, outputFile) + if err != nil { + return err + } + pub.ContentKey = encryptionKey + // calculate the output file size and checksum + stats, err := outputFile.Stat() + if err == nil && (stats.Size() > 0) { + filesize := stats.Size() + pub.Size = filesize + cs := checksum(outputFile) + pub.Checksum = cs + } + if stats.Size() == 0 { + return errors.New("empty output file") + } + return nil +} + +// processPDF wraps a PDF file inside a Readium Package and encrypts its resources +func processPDF(pub *apilcp.LcpPublication, inputPath string, outputPath string, encrypter crypto.Encrypter) error { + + // generate a temp Readium Package (rwpp) which embeds the PDF file; its title is the PDF file name + tmpPackagePath := outputPath + ".tmp" + err := pack.BuildRPFFromPDF(filepath.Base(inputPath), inputPath, tmpPackagePath) + // will remove the tmp file even if an error is returned + defer os.Remove(tmpPackagePath) + // process error + if err != nil { + return err + } + + // build an encrypted package + return buildEncryptedRPF(pub, tmpPackagePath, outputPath, encrypter) +} + +// processLPF transforms a W3C LPF file into a Readium Package and encrypts its resources +func processLPF(pub *apilcp.LcpPublication, inputPath string, outputPath string, encrypter crypto.Encrypter) error { + + // generate a tmp Readium Package (rwpp) out of a W3C Package (lpf) + tmpPackagePath := outputPath + ".tmp" + err := pack.BuildRPFFromLPF(inputPath, tmpPackagePath) + // will remove the tmp file even if an error is returned + defer os.Remove(tmpPackagePath) + // process error + if err != nil { + return err + } + + // build an encrypted package + return buildEncryptedRPF(pub, tmpPackagePath, outputPath, encrypter) +} + +// processRPF encrypts the source Readium Package +func processRPF(pub *apilcp.LcpPublication, inputPath string, outputPath string, encrypter crypto.Encrypter) error { + + // build an encrypted package + return buildEncryptedRPF(pub, inputPath, outputPath, encrypter) +} + +// buildEncryptedRPF builds an encrypted Readium package out of an un-encrypted one +// FIXME: it cannot be used for EPUB as long as Do() and Process() are not merged +func buildEncryptedRPF(pub *apilcp.LcpPublication, inputPath string, outputPath string, encrypter crypto.Encrypter) error { + + // create a reader on the un-encrypted readium package + reader, err := pack.OpenRPF(inputPath) + if err != nil { + return err + } + // create the encrypted package file + outputFile, err := os.Create(outputPath) + if err != nil { + return err + } + defer outputFile.Close() + // create a writer on the encrypted package + writer, err := reader.NewWriter(outputFile) + if err != nil { + return err + } + // encrypt resources from the input package, return the encryption key + encryptionKey, err := pack.Process(encrypter, reader, writer) + if err != nil { + return err + } + pub.ContentKey = encryptionKey + + err = writer.Close() + if err != nil { + return err + } + + // calculate the output file size and checksum + stats, err := outputFile.Stat() + if err == nil && (stats.Size() > 0) { + filesize := stats.Size() + pub.Size = filesize + cs := checksum(outputFile) + pub.Checksum = cs + } + if stats.Size() == 0 { + return errors.New("empty output file") + } + return nil +} + +// NotifyLcpServer notifies the License Server of the encryption of newly added publication +func NotifyLcpServer(pub *apilcp.LcpPublication, licenseServerURL string, username string, password string) error { + + // No license server URL is not an error, simply a silent encryption + if licenseServerURL == "" { + fmt.Println("No notification sent to the License Server") + return nil + } + // prepare the call to service/content/, + var urlBuffer bytes.Buffer + urlBuffer.WriteString(licenseServerURL) + urlBuffer.WriteString("/contents/") + urlBuffer.WriteString(pub.ContentID) + + jsonBody, err := json.Marshal(*pub) + if err != nil { + return err + } + req, err := http.NewRequest("PUT", urlBuffer.String(), bytes.NewReader(jsonBody)) + if err != nil { + return err + } + req.SetBasicAuth(username, password) + client := &http.Client{} + resp, err := client.Do(req) + if err != nil { + return err + } + if (resp.StatusCode != 302) && (resp.StatusCode/100) != 2 { //302=found or 20x reply = OK + return fmt.Errorf("lcp server error %d", resp.StatusCode) + } + + return nil +} diff --git a/encrypt/store_publication.go b/encrypt/store_publication.go new file mode 100644 index 00000000..3f91db19 --- /dev/null +++ b/encrypt/store_publication.go @@ -0,0 +1,59 @@ +// Copyright 2021 Readium Foundation. All rights reserved. +// Use of this source code is governed by a BSD-style license +// that can be found in the LICENSE file exposed on Github (readium) in the project repository. + +package encrypt + +import ( + "errors" + "os" + "strings" + + apilcp "github.com/readium/readium-lcp-server/lcpserver/api" + "github.com/readium/readium-lcp-server/storage" +) + +// StorePublication stores an encrypted file into its definitive storage. +// Only called for S3 buckets. +func StorePublication(pub *apilcp.LcpPublication, inputPath string, storagePath string) error { + + if pub.StorageMode != apilcp.Storage_s3 { + return errors.New("only S3 storage is processed in StorePublication") + } + + s3Split := strings.Split(storagePath, ":") + + s3conf := storage.S3Config{} + s3conf.Region = s3Split[1] + s3conf.Bucket = s3Split[2] + + var store storage.Store + // init the S3 storage + store, err := storage.S3(s3conf) + if err != nil { + return errors.New("could not init the S3 storage") + } + + // open the encrypted file, defer its deletion + file, err := os.Open(inputPath) + if err != nil { + return err + } + defer cleanupTempFile(file) + + // add the file to the storage, named by contentID, without file extension + _, err = store.Add(pub.ContentID, file) + if err != nil { + return err + } + return nil +} + +// cleanupTempFile closes and deletes a temporary file +func cleanupTempFile(f *os.File) { + if f == nil { + return + } + f.Close() + os.Remove(f.Name()) +} diff --git a/frontend/webpublication/webpublication.go b/frontend/webpublication/webpublication.go index a9a45ede..a19564cf 100644 --- a/frontend/webpublication/webpublication.go +++ b/frontend/webpublication/webpublication.go @@ -5,31 +5,19 @@ package webpublication import ( - "bytes" "database/sql" - "encoding/json" "errors" "io" "io/ioutil" "log" "mime/multipart" - "net/http" "os" - "path" "path/filepath" "strings" - "time" - "github.com/readium/readium-lcp-server/api" "github.com/readium/readium-lcp-server/config" - "github.com/readium/readium-lcp-server/epub" - "github.com/readium/readium-lcp-server/lcpencrypt/encrypt" + "github.com/readium/readium-lcp-server/encrypt" apilcp "github.com/readium/readium-lcp-server/lcpserver/api" - "github.com/readium/readium-lcp-server/license" - "github.com/readium/readium-lcp-server/pack" - uuid "github.com/satori/go.uuid" - - "github.com/Machiel/slugify" ) // Publication status @@ -79,7 +67,7 @@ func (pubManager PublicationManager) Get(id int64) (Publication, error) { } defer dbGetByID.Close() - records, err := dbGetByID.Query(id) + records, _ := dbGetByID.Query(id) if records.Next() { var pub Publication err = records.Scan( @@ -103,7 +91,7 @@ func (pubManager PublicationManager) GetByUUID(uuid string) (Publication, error) } defer dbGetByUUID.Close() - records, err := dbGetByUUID.Query(uuid) + records, _ := dbGetByUUID.Query(uuid) if records.Next() { var pub Publication err = records.Scan( @@ -127,7 +115,7 @@ func (pubManager PublicationManager) CheckByTitle(title string) (int64, error) { } defer dbGetByTitle.Close() - records, err := dbGetByTitle.Query(title) + records, _ := dbGetByTitle.Query(title) if records.Next() { var res int64 err = records.Scan(&res) @@ -140,144 +128,33 @@ func (pubManager PublicationManager) CheckByTitle(title string) (int64, error) { return -1, ErrNotFound } -// encryptPublication encrypts an EPUB, PDF, LPF or RPF file and provides the resulting file to the LCP server +// encryptPublication encrypts a publication, notifies the License Server +// and inserts a record in the database. func encryptPublication(inputPath string, pub Publication, pubManager PublicationManager) error { - // generate a new uuid; this will be the content id in the lcp server - uid, err := uuid.NewV4() - if err != nil { - return err - } - contentUUID := uid.String() - if len(pub.UUID) > 0 { - contentUUID = pub.UUID - } - - // set the encryption profile from the config file - var lcpProfile license.EncryptionProfile - if pubManager.config.Profile == "1.0" { - lcpProfile = license.V1Profile - } else { - lcpProfile = license.BasicProfile - } - - // create a temp file in the frontend "encrypted repository" - outputFilename := contentUUID + ".tmp" - outputPath := path.Join(pubManager.config.FrontendServer.EncryptedRepository, outputFilename) - - // encrypt the master file found at inputPath, write in the temp file, in the "encrypted repository" - var encryptedPub encrypt.EncryptionArtifact - var contentType string - - switch filepath.Ext(inputPath) { - // process EPUB files - case ".epub": - contentType = epub.ContentType_EPUB - encryptedPub, err = encrypt.EncryptEpub(inputPath, outputPath) - - // process PDF files - case ".pdf": - contentType = "application/pdf+lcp" - clearWebPubPath := outputPath + ".webpub" - err = pack.BuildRPFFromPDF(pub.Title, inputPath, clearWebPubPath) - if err != nil { - log.Printf("Error building webpub package: %s", err) - return err - } - defer os.Remove(clearWebPubPath) - encryptedPub, err = encrypt.EncryptPackage(lcpProfile, clearWebPubPath, outputPath) - - // process LPF files - case ".lpf": - // FIXME: short term solution; should be extended to other profiles - contentType = "application/audiobook+lcp" - clearWebPubPath := outputPath + ".webpub" - err = pack.BuildRPFFromLPF(inputPath, clearWebPubPath) - if err != nil { - return err - } - defer os.Remove(clearWebPubPath) - encryptedPub, err = encrypt.EncryptPackage(lcpProfile, clearWebPubPath, outputPath) - - // process RPF Audiobook files - case ".audiobook": - contentType = "application/audiobook+lcp" - encryptedPub, err = encrypt.EncryptPackage(lcpProfile, inputPath, outputPath) - - // process RPF Divina files - case ".divina": - contentType = "application/divina+lcp" - encryptedPub, err = encrypt.EncryptPackage(lcpProfile, inputPath, outputPath) - - // process RPF PDF files - case ".rpf": - contentType = "application/pdf+lcp" - encryptedPub, err = encrypt.EncryptPackage(lcpProfile, inputPath, outputPath) - - // unknown file - default: - return errors.New("Could not match the file extension: " + inputPath) - } - - if err != nil { - // unable to encrypt the master file - if _, statErr := os.Stat(inputPath); statErr == nil { - os.Remove(inputPath) - } - return err - } + var notification *apilcp.LcpPublication - // prepare the import request to the lcp server - contentDisposition := slugify.Slugify(pub.Title) - lcpPublication := apilcp.LcpPublication{} - lcpPublication.ContentID = contentUUID - lcpPublication.ContentKey = encryptedPub.EncryptionKey - // both frontend and lcp server must understand this path (warning if using Docker containers) - lcpPublication.Output = outputPath - lcpPublication.ContentDisposition = &contentDisposition - lcpPublication.Checksum = &encryptedPub.Checksum - lcpPublication.Size = &encryptedPub.Size - lcpPublication.ContentType = contentType - - // json encode the payload - jsonBody, err := json.Marshal(lcpPublication) + // encrypt the publication + // FIXME: work on a direct storage of the output file. + outputRepo := pubManager.config.FrontendServer.EncryptedRepository + notification, err := encrypt.ProcessPublication("", inputPath, "", outputRepo, "", "") if err != nil { return err } - // send the content to the LCP server - lcpServerConfig := pubManager.config.LcpServer - lcpURL := lcpServerConfig.PublicBaseUrl + "/contents/" + contentUUID - log.Println("PUT " + lcpURL) - req, err := http.NewRequest("PUT", lcpURL, bytes.NewReader(jsonBody)) - if err != nil { - return err - } - // authenticate - lcpUpdateAuth := pubManager.config.LcpUpdateAuth - if pubManager.config.LcpUpdateAuth.Username != "" { - req.SetBasicAuth(lcpUpdateAuth.Username, lcpUpdateAuth.Password) - } - // set the payload type - req.Header.Add("Content-Type", api.ContentType_LCP_JSON) - - var lcpClient = &http.Client{ - Timeout: time.Second * 60, - } - // sends the import request to the lcp server - resp, err := lcpClient.Do(req) + // send a notification to the License server + err = encrypt.NotifyLcpServer( + notification, + pubManager.config.LcpServer.PublicBaseUrl, + pubManager.config.LcpUpdateAuth.Username, + pubManager.config.LcpUpdateAuth.Password) if err != nil { return err } - if resp.StatusCode != 201 { - // error on creation - return err - } - // store the new publication in the db // the publication uuid is the lcp db content id. - pub.UUID = contentUUID + pub.UUID = notification.ContentID pub.Status = StatusOk dbAdd, err := pubManager.db.Prepare("INSERT INTO publication (uuid, title, status) VALUES ( ?, ?, ?)") if err != nil { @@ -289,40 +166,49 @@ func encryptPublication(inputPath string, pub Publication, pubManager Publicatio pub.UUID, pub.Title, pub.Status) + return err } // Add adds a new publication -// Encrypts a master File and sends the content to the LCP server +// Encrypts a master File and notifies the License server func (pubManager PublicationManager) Add(pub Publication) error { // get the path to the master file - inputPath := path.Join( + inputPath := filepath.Join( pubManager.config.FrontendServer.MasterRepository, pub.MasterFilename) - log.Println("Add a publication from path " + inputPath) - if _, err := os.Stat(inputPath); err != nil { // the master file does not exist return err } - // encrypt the publication and send the content to the LCP server - return encryptPublication(inputPath, pub, pubManager) + + // encrypt the publication and send a notification to the License server + err := encryptPublication(inputPath, pub, pubManager) + if err != nil { + return err + } + + // delete the master file + err = os.Remove(inputPath) + if err != nil { + return err + } + + return nil } // Upload creates a new publication, named after a POST form parameter. -// The file is processed, encrypted and sent to the LCP server. +// Encrypts a master File and notifies the License server func (pubManager PublicationManager) Upload(file multipart.File, extension string, pub Publication) error { - // create a temp file + // create a temp file in the default directory tmpfile, err := ioutil.TempFile("", "uploaded-*"+extension) if err != nil { return err } defer os.Remove(tmpfile.Name()) - log.Println("Upload a publication, use a tmp file") - // copy the request payload to the temp file if _, err = io.Copy(tmpfile, file); err != nil { return err @@ -333,13 +219,8 @@ func (pubManager PublicationManager) Upload(file multipart.File, extension strin return err } - // process and encrypt the publication, send the content to the LCP server - err = encryptPublication(tmpfile.Name(), pub, pubManager) - if err != nil { - return err - } - - return nil + // encrypt the publication and send a notification to the License server + return encryptPublication(tmpfile.Name(), pub, pubManager) } // Update updates a publication @@ -382,17 +263,6 @@ func (pubManager PublicationManager) Delete(id int64) error { if err != nil { return err } - - // delete the epub file from the master repository - // FIXME: make it work for all kinds of extensions - inputPath := path.Join(pubManager.config.FrontendServer.MasterRepository, title+".epub") - - if _, err := os.Stat(inputPath); err == nil { - err = os.Remove(inputPath) - if err != nil { - return err - } - } } result.Close() diff --git a/lcpencrypt/encrypt/encrypt.go b/lcpencrypt/encrypt/encrypt.go deleted file mode 100644 index 7497c3bd..00000000 --- a/lcpencrypt/encrypt/encrypt.go +++ /dev/null @@ -1,140 +0,0 @@ -// Copyright 2020 Readium Foundation. All rights reserved. -// Use of this source code is governed by a BSD-style license -// that can be found in the LICENSE file exposed on Github (readium) in the project repository. - -package encrypt - -import ( - "archive/zip" - "crypto/sha256" - "encoding/hex" - "errors" - "io" - "os" - - "github.com/readium/readium-lcp-server/crypto" - "github.com/readium/readium-lcp-server/epub" - "github.com/readium/readium-lcp-server/license" - "github.com/readium/readium-lcp-server/pack" -) - -// EncryptionArtifact is the result of a successful encryption process -type EncryptionArtifact struct { - // The encryption process will have put the resulting encrypted file at this place. - // It is the caller's responsibility to handle it afterwards - Path string - // The encryption key that was randomly generated to encrypt the file. - EncryptionKey []byte - // The size of the resulting file - Size int64 - // A Hex-Encoded SHA256 checksum of the encrypted package - Checksum string -} - -func encryptionError(message string) (EncryptionArtifact, error) { - return EncryptionArtifact{}, errors.New(message) -} - -// EncryptPackage generates an encrypted output RPF out of the input RPF -// It is called from the test frontend server -func EncryptPackage(profile license.EncryptionProfile, inputPath string, outputPath string) (EncryptionArtifact, error) { - - // create an AES encrypter for publication resources - encrypter := crypto.NewAESEncrypter_PUBLICATION_RESOURCES() - - // create a reader on the un-encrypted readium package - reader, err := pack.OpenRPF(inputPath) - if err != nil { - return encryptionError(err.Error()) - } - - // create the encrypted package file - outputFile, err := os.Create(outputPath) - if err != nil { - return encryptionError("Unable to create output file") - } - defer outputFile.Close() - - // create a writer on the encrypted package - writer, err := reader.NewWriter(outputFile) - if err != nil { - return encryptionError("Unable to create output writer") - } - - // encrypt resources from the input package, return the encryption key - encryptionKey, err := pack.Process(profile, encrypter, reader, writer) - if err != nil { - return encryptionError("Unable to encrypt file") - } - - err = writer.Close() - if err != nil { - return encryptionError("Unable to close the writer") - } - - // calculate the output file size and checksum - hasher := sha256.New() - outputFile.Seek(0, 0) - size, err := io.Copy(hasher, outputFile) - if err != nil { - return encryptionError("Could not generate a hash for the file") - } - - return EncryptionArtifact{ - Path: outputPath, - EncryptionKey: encryptionKey, - Size: size, - Checksum: hex.EncodeToString(hasher.Sum(nil)), - }, nil -} - -// EncryptEpub generates an encrypted output file out of the input file -// It is called from the test frontend server; inputPath is therefore a file path. -func EncryptEpub(inputPath string, outputPath string) (EncryptionArtifact, error) { - - if _, err := os.Stat(inputPath); err != nil { - return encryptionError("Input file does not exist") - } - - // create a zip reader from the input path - zr, err := zip.OpenReader(inputPath) - if err != nil { - return encryptionError("Unable to open the input file") - } - defer zr.Close() - - // parse the source epub - epubContent, err := epub.Read(&zr.Reader) - if err != nil { - return encryptionError("Error reading epub content") - } - - // create the output file - outputFile, err := os.Create(outputPath) - if err != nil { - return encryptionError("Unable to create output file") - } - defer outputFile.Close() - - // pack / encrypt the epub content, fill the output file - encrypter := crypto.NewAESEncrypter_PUBLICATION_RESOURCES() - _, encryptionKey, err := pack.Do(encrypter, epubContent, outputFile) - if err != nil { - return encryptionError("Unable to encrypt file") - } - - // calculate the output file size and checksum - hasher := sha256.New() - outputFile.Seek(0, 0) - size, err := io.Copy(hasher, outputFile) - if err != nil { - return encryptionError("Could not generate a hash for the file") - } - - return EncryptionArtifact{ - Path: outputPath, - EncryptionKey: encryptionKey, - Size: size, - Checksum: hex.EncodeToString(hasher.Sum(nil)), - }, nil -} diff --git a/lcpencrypt/encrypt/encrypt_test.go b/lcpencrypt/encrypt/encrypt_test.go deleted file mode 100644 index 0c67ca95..00000000 --- a/lcpencrypt/encrypt/encrypt_test.go +++ /dev/null @@ -1,38 +0,0 @@ -// Copyright 2020 Readium Foundation. All rights reserved. -// Use of this source code is governed by a BSD-style license -// that can be found in the LICENSE file exposed on Github (readium) in the project repository. - -package encrypt - -import ( - "fmt" - "testing" - - "github.com/readium/readium-lcp-server/license" -) - -func TestEncryptEPUB(t *testing.T) { - - inputPath := "../../test/samples/sample.epub" - outputPath := "../../test/samples/sample-encrypted.epub" - result, err := EncryptEpub(inputPath, outputPath) - if err != nil { - t.Error(err.Error()) - } - - fmt.Printf("output: %s size %d\n", result.Path, result.Size) - -} - -func TestEncryptRPF(t *testing.T) { - - inputPath := "../../test/samples/tst-features.divina" - outputPath := "../../test/samples/tst-features-encrypted.divina" - result, err := EncryptPackage(license.BasicProfile, inputPath, outputPath) - if err != nil { - t.Error(err.Error()) - } - - fmt.Printf("output: %s size %d\n", result.Path, result.Size) - -} diff --git a/lcpencrypt/lcpencrypt.go b/lcpencrypt/lcpencrypt.go index a2aff6fd..7dec99db 100644 --- a/lcpencrypt/lcpencrypt.go +++ b/lcpencrypt/lcpencrypt.go @@ -5,311 +5,49 @@ package main import ( - "archive/zip" - "bytes" - "crypto/sha256" - "encoding/hex" "encoding/json" + "errors" "flag" "fmt" - "io" - "log" - "net/http" "os" - "path/filepath" - "strconv" - "strings" - "github.com/readium/readium-lcp-server/crypto" - "github.com/readium/readium-lcp-server/epub" - apilcp "github.com/readium/readium-lcp-server/lcpserver/api" - "github.com/readium/readium-lcp-server/license" - "github.com/readium/readium-lcp-server/pack" - uuid "github.com/satori/go.uuid" + "github.com/readium/readium-lcp-server/encrypt" ) -// notification of newly added content (Publication) -func notifyLcpServer(lcpService, contentid string, lcpPublication apilcp.LcpPublication, username string, password string) error { - - //exchange encryption key with lcp service/content/, - //Payload: - // content-id: unique id for the content - // content-encryption-key: encryption key used for the content - // protected-content-location: full path of the encrypted file - // protected-content-length: content length in bytes - // protected-content-sha256: content sha - // protected-content-disposition: encrypted file name - // protected-content-type: encrypted file content type - //fmt.Printf("lcpsv = %s\n", *lcpsv) - var urlBuffer bytes.Buffer - urlBuffer.WriteString(lcpService) - urlBuffer.WriteString("/contents/") - urlBuffer.WriteString(contentid) - - jsonBody, err := json.Marshal(lcpPublication) - if err != nil { - return err - } - req, err := http.NewRequest("PUT", urlBuffer.String(), bytes.NewReader(jsonBody)) - if err != nil { - return err - } - req.SetBasicAuth(username, password) - client := &http.Client{} - resp, err := client.Do(req) - if err != nil { - return err - } - if (resp.StatusCode != 302) && (resp.StatusCode/100) != 2 { //302=found or 20x reply = OK - return fmt.Errorf("lcp server error %d", resp.StatusCode) - } - - return nil -} - +// showHelpAndExit displays some help and exits. func showHelpAndExit() { - log.Println("lcpencrypt protects a publication using the LCP DRM") - log.Println("-input source file path") - log.Println("[-profile] encryption profile") - log.Println("[-contentid] optional content identifier, if omitted a new one will be generated") - log.Println("[-output] optional target location for protected content (file system or http PUT)") - log.Println("[-lcpsv] optional http endpoint for the License server") - log.Println("[-login] login ( needed for License server) ") - log.Println("[-password] password ( needed for License server)") - log.Println("[-help] : help information") + fmt.Println("lcpencrypt protects a publication using the LCP DRM") + fmt.Println("-input source epub/pdf/lpf file locator (file system or http GET)") + fmt.Println("[-output] optional, target path of encrypted publications") + fmt.Println("[-temp] optional, working folder for temporary files") + fmt.Println("[-storage] optional, final storage of the encrypted publication, fs or s3") + fmt.Println("[-url] optional, base url associated with the storagen") + fmt.Println("[-contentid] optional, content identifier; if omitted a uuid is generated") + fmt.Println("[-lcpsv] optional, http endpoint, notification of the License server") + fmt.Println("[-login] login (License server) ") + fmt.Println("[-password] password (License server)") + fmt.Println("[-help] : help information") os.Exit(0) - return -} - -func exitWithError(lcpPublication apilcp.LcpPublication, err error, errorlevel int) { - - os.Stdout.WriteString(lcpPublication.ErrorMessage + "; level " + strconv.Itoa(errorlevel)) - os.Stdout.WriteString("\n") - if err != nil { - os.Stdout.WriteString(err.Error()) - } - os.Stdout.WriteString("\n") - /* kept for future debug - jsonBody, err := json.MarshalIndent(lcpPublication, " ", " ") - if err != nil { - os.Stdout.WriteString("Error creating json lcpPublication\n") - os.Exit(errorlevel) - } - os.Stdout.Write(jsonBody) - os.Stdout.WriteString("\n") - */ - os.Exit(errorlevel) -} - -// checksum calculates the checksum of a file -func checksum(file *os.File) string { - - hasher := sha256.New() - file.Seek(0, 0) - if _, err := io.Copy(hasher, file); err != nil { - return "" - } - return hex.EncodeToString(hasher.Sum(nil)) -} - -func outputExtension(sourceExt string) string { - - var targetExt string - switch sourceExt { - case ".epub": - // an LCP protected EPUB file keeps the same extension - targetExt = ".epub" - case ".pdf": - targetExt = ".lcpdf" - case ".rpf": - // short term solution. We'll need to inspect the manifest and check conformsTo, - // to be certain this package contains a pdf - targetExt = ".lcpdf" - case ".audiobook": - targetExt = ".lcpau" - case ".divina": - targetExt = ".lcpdi" - case ".lpf": - // short term solution. We'll need to inspect the manifest and check conformsTo, - // to be certain this is an audiobook (vs another profile of Web Publication) - targetExt = ".lcpau" - } - return targetExt -} - -// buildEncryptedRPF builds an encrypted Readium package out of an un-encrypted one -// FIXME: it cannot be used for EPUB as long as Do() and Process() are not merged -func buildEncryptedRPF(pub *apilcp.LcpPublication, inputPath string, encrypter crypto.Encrypter, lcpProfile license.EncryptionProfile) error { - - // create a reader on the un-encrypted readium package - reader, err := pack.OpenRPF(inputPath) - if err != nil { - pub.ErrorMessage = "Error opening package " + inputPath - return err - } - // create the encrypted package file - outputFile, err := os.Create(pub.Output) - if err != nil { - pub.ErrorMessage = "Error creating the output package" - return err - } - defer outputFile.Close() - // create a writer on the encrypted package - writer, err := reader.NewWriter(outputFile) - if err != nil { - pub.ErrorMessage = "Error opening the output package" - return err - } - // encrypt resources from the input package, return the encryption key - encryptionKey, err := pack.Process(lcpProfile, encrypter, reader, writer) - if err != nil { - pub.ErrorMessage = "Error encrypting the publication" - return err - } - pub.ContentKey = encryptionKey - - err = writer.Close() - if err != nil { - pub.ErrorMessage = "Unable to close the writer" - return err - } - - // calculate the output file size and checksum - stats, err := outputFile.Stat() - if err == nil && (stats.Size() > 0) { - filesize := stats.Size() - pub.Size = &filesize - cs := checksum(outputFile) - pub.Checksum = &cs - } - if stats.Size() == 0 { - pub.ErrorMessage = "Empty output file" - return err - } - return nil -} - -// processEPUB encrypts resources in an EPUB -func processEPUB(pub *apilcp.LcpPublication, inputPath string, encrypter crypto.Encrypter) error { - - pub.ContentType = epub.ContentType_EPUB - - // create a zip reader from the input path - zr, err := zip.OpenReader(inputPath) - if err != nil { - pub.ErrorMessage = "Error opening the epub file" - return err - } - defer zr.Close() - - // generate an EPUB object - epub, err := epub.Read(&zr.Reader) - if err != nil { - pub.ErrorMessage = "Error reading epub content" - return err - } - // create the output file - outputFile, err := os.Create(pub.Output) - if err != nil { - pub.ErrorMessage = "Error writing output file" - return err - } - // will close the output file - defer outputFile.Close() - // encrypt the content of the publication, - // write them into the output file - _, encryptionKey, err := pack.Do(encrypter, epub, outputFile) - if err != nil { - pub.ErrorMessage = "Error encrypting the EPUB content" - return err - } - pub.ContentKey = encryptionKey - // calculate the output file size and checksum - stats, err := outputFile.Stat() - if err == nil && (stats.Size() > 0) { - filesize := stats.Size() - pub.Size = &filesize - cs := checksum(outputFile) - pub.Checksum = &cs - } - if stats.Size() == 0 { - pub.ErrorMessage = "Empty output file" - return err - } - return nil -} - -// processPDF wraps an encrypted PDF file inside a Readium package -func processPDF(pub *apilcp.LcpPublication, inputPath string, encrypter crypto.Encrypter, lcpProfile license.EncryptionProfile) error { - - pub.ContentType = "application/pdf+lcp" - - // generate a temp Readium Package (rwpp) which embeds the PDF file; its title is the PDF file name - tmpPackagePath := pub.Output + ".tmp" - err := pack.BuildRPFFromPDF(filepath.Base(inputPath), inputPath, tmpPackagePath) - if err != nil { - pub.ErrorMessage = "Error building Web Publication package from PDF" - return err - } - defer os.Remove(tmpPackagePath) - - // build an encrypted package - err = buildEncryptedRPF(pub, tmpPackagePath, encrypter, lcpProfile) - return err } -// processLPF transforms a W3C LPF file into a Readium Package and encrypts its resources -func processLPF(pub *apilcp.LcpPublication, inputPath string, encrypter crypto.Encrypter, lcpProfile license.EncryptionProfile, outputExt string) error { - - // When other kinds of LPF files are created, a switch on outputExt will be used - // to select the proper mime-type - pub.ContentType = "application/audiobook+lcp" +// exitWithError outputs an error message and exits. +func exitWithError(context string, err error) { - // generate a tmp Readium Package (rwpp) out of W3C Package (lpf) - tmpPackagePath := pub.Output + ".webpub" - err := pack.BuildRPFFromLPF(inputPath, tmpPackagePath) - // will remove the tmp file even if an error is returned - defer os.Remove(tmpPackagePath) - // process error - if err != nil { - pub.ErrorMessage = "Error building RPF from LPF" - return err - } - - // build an encrypted package - err = buildEncryptedRPF(pub, tmpPackagePath, encrypter, lcpProfile) - return err -} - -// processRPF encrypts the source Readium Package -func processRPF(pub *apilcp.LcpPublication, inputPath string, encrypter crypto.Encrypter, lcpProfile license.EncryptionProfile, outputExt string) error { - - // select a mime-type - switch outputExt { - case ".lcpau": - pub.ContentType = "application/audiobook+lcp" - case ".lcpdi": - pub.ContentType = "application/divina+lcp" - case ".lcpdf": - pub.ContentType = "application/pdf+lcp" - } - - // build an encrypted package - err := buildEncryptedRPF(pub, inputPath, encrypter, lcpProfile) - return err + fmt.Println(context, ":", err.Error()) + os.Exit(1) } func main() { - var err error - var pub apilcp.LcpPublication var inputPath = flag.String("input", "", "source epub/pdf/lpf file locator (file system or http GET)") - var contentid = flag.String("contentid", "", "optional content identifier; if omitted a new uuid is generated") - var outputFilename = flag.String("output", "", "optional target location for the encrypted content (file system or http PUT)") - var lcpsv = flag.String("lcpsv", "", "optional http endpoint of the License server (adds content)") + var outputRepo = flag.String("output", "", "optional, target folder of encrypted publications") + var tempRepo = flag.String("temp", "", "optional, working folder for temporary files") + var storageRepo = flag.String("storage", "", "optional, final storage of the encrypted publication, fs or s3") + var storageURL = flag.String("url", "", "optional, base url associated with the storage") + var contentid = flag.String("contentid", "", "optional, content identifier; if omitted a uuid is generated") + var lcpsv = flag.String("lcpsv", "", "optional, http endpoint, notification of the License server") var username = flag.String("login", "", "login (License server)") var password = flag.String("password", "", "password (License server)") - var profile = flag.String("profile", "basic", "LCP Profile to use for encryption: 'basic' or 'v1'") var help = flag.Bool("help", false, "shows information") @@ -321,88 +59,31 @@ func main() { } if *lcpsv != "" && (*username == "" || *password == "") { - pub.ErrorMessage = "incorrect parameters, lcpsv needs a login and password, for more information type 'lcpencrypt -help' " - exitWithError(pub, nil, 10) - } - - if *contentid == "" { // contentID not set -> generate a new one - uid, err := uuid.NewV4() - if err != nil { - exitWithError(pub, err, 20) - } - *contentid = uid.String() - } - pub.ContentID = *contentid - - // if the output file name not set, - // then [content-id].[ext] is created into the working directory - inputExt := filepath.Ext(*inputPath) - var basefilename string - var outputExt string - if *outputFilename == "" { - workingDir, _ := os.Getwd() - outputExt = outputExtension(inputExt) - *outputFilename = strings.Join([]string{workingDir, string(os.PathSeparator), *contentid, outputExt}, "") - basefilename = filepath.Base(*inputPath) - } else { - outputExt = filepath.Ext(*outputFilename) - basefilename = filepath.Base(*outputFilename) + exitWithError("Parameters", errors.New("incorrect parameters, lcpsv needs a login and password, for more information type 'lcpencrypt -help' ")) } - pub.ContentDisposition = &basefilename - // reminder: the output path must be accessible from the license server - pub.Output = *outputFilename - - var lcpProfile license.EncryptionProfile - if *profile == "v1" { - lcpProfile = license.V1Profile - } else { // covers missing parameter - lcpProfile = license.BasicProfile + if *storageRepo != "" && *storageURL == "" { + exitWithError("Parameters", errors.New("incorrect parameters, storage requires url, for more information type 'lcpencrypt -help' ")) } - encrypter := crypto.NewAESEncrypter_PUBLICATION_RESOURCES() - - // select the encryption process - switch inputExt { - case ".epub": - err := processEPUB(&pub, *inputPath, encrypter) - if err != nil { - exitWithError(pub, err, 30) - } - case ".pdf": - err := processPDF(&pub, *inputPath, encrypter, lcpProfile) - if err != nil { - exitWithError(pub, err, 31) - } - case ".lpf": - err := processLPF(&pub, *inputPath, encrypter, lcpProfile, outputExt) - if err != nil { - exitWithError(pub, err, 32) - } - case ".audiobook", ".divina", ".rpf": - err := processRPF(&pub, *inputPath, encrypter, lcpProfile, outputExt) - if err != nil { - exitWithError(pub, err, 33) - } + // encrypt the publication + pub, err := encrypt.ProcessPublication(*contentid, *inputPath, *tempRepo, *outputRepo, *storageRepo, *storageURL) + if err != nil { + exitWithError("Process a publication", err) } - // notify the LCP Server - if *lcpsv != "" { - err = notifyLcpServer(*lcpsv, *contentid, pub, *username, *password) - if err != nil { - pub.ErrorMessage = "Error notifying the License Server" - exitWithError(pub, err, 40) - } else { - os.Stdout.WriteString("License Server was notified\n") - } + // notify the license server + err = encrypt.NotifyLcpServer(pub, *lcpsv, *username, *password) + if err != nil { + exitWithError("Notify the LCP Server", err) } // write a json message to stdout for debug purpose jsonBody, err := json.MarshalIndent(pub, " ", " ") if err != nil { - pub.ErrorMessage = "Error creating json pub" - exitWithError(pub, err, 50) + exitWithError("Debug Message", errors.New("JSON error")) } + fmt.Println("Encryption message:") os.Stdout.Write(jsonBody) - os.Stdout.WriteString("\nEncryption was successful\n") + fmt.Println("\nEncryption was successful)") os.Exit(0) } diff --git a/lcpserver/api/license.go b/lcpserver/api/license.go index 4279db3a..e61889e7 100644 --- a/lcpserver/api/license.go +++ b/lcpserver/api/license.go @@ -30,13 +30,13 @@ import ( ) // ErrMandatoryInfoMissing sets an error message returned to the caller -var ErrMandatoryInfoMissing = errors.New("Mandatory info missing in the input body") +var ErrMandatoryInfoMissing = errors.New("mandatory info missing in the input body") // ErrBadHexValue sets an error message returned to the caller -var ErrBadHexValue = errors.New("Erroneous user_key.hex_value can't be decoded") +var ErrBadHexValue = errors.New("erroneous user_key.hex_value can't be decoded") // ErrBadValue sets an error message returned to the caller -var ErrBadValue = errors.New("Erroneous user_key.value, can't be decoded") +var ErrBadValue = errors.New("erroneous user_key.value, can't be decoded") // checkGetLicenseInput: if we generate or get a license, check mandatory information in the input body // and compute request parameters @@ -208,6 +208,9 @@ func buildLicensedPublication(lic *license.License, s Server) (buf bytes.Buffer, zipWriter := zip.NewWriter(&buf) err = copyZipFiles(zipWriter, zr) + if err != nil { + return buf, err + } // Encode the license to JSON, remove the trailing newline // write the buffer in the zip @@ -323,8 +326,6 @@ func GenerateLicense(w http.ResponseWriter, r *http.Request, s Server) { // get the content id from the request URL contentID := vars["content_id"] - log.Println("Generate License for content id", contentID) - // get the input body // note: no need to create licIn / licOut here, as the input body contains // info that we want to keep in the full license. @@ -360,6 +361,9 @@ func GenerateLicense(w http.ResponseWriter, r *http.Request, s Server) { //problem.Error(w, r, problem.Problem{Detail: err.Error(), Instance: contentID}, http.StatusInternalServerError) return } + + log.Println("New License:", lic.ID, ". Content:", contentID, "User:", lic.User.ID) + // set http headers w.Header().Add("Content-Type", api.ContentType_LCP_JSON) w.Header().Add("Content-Disposition", `attachment; filename="license.lcpl"`) @@ -772,20 +776,6 @@ func notifyLsdServer(l license.License, s Server) { } else { defer req.Body.Close() _ = s.Licenses().UpdateLsdStatus(l.ID, int32(response.StatusCode)) - // message to the console - log.Println("Notify Lsd Server of a new License with id " + l.ID + " = " + strconv.Itoa(response.StatusCode)) } } } - -// utility: log a license for debug purposes -// ex: logLicense("build licence:", licOut) -func logLicense(msg string, l *license.License) { - log.Println(msg) - jsonBody, errj := json.Marshal(*l) - if errj != nil { - log.Println("logLicense: not well formed json") - return - } - log.Print(string(jsonBody)) -} diff --git a/lcpserver/api/store.go b/lcpserver/api/store.go index 4c00ac13..42185b10 100644 --- a/lcpserver/api/store.go +++ b/lcpserver/api/store.go @@ -34,18 +34,24 @@ type Server interface { Source() *pack.ManualSource } -// LcpPublication is a struct for communication with lcp-server +// LcpPublication is used for communication with the License Server type LcpPublication struct { - ContentID string `json:"content-id"` - ContentKey []byte `json:"content-encryption-key"` - Output string `json:"protected-content-location"` - Size *int64 `json:"protected-content-length"` - Checksum *string `json:"protected-content-sha256"` - ContentDisposition *string `json:"protected-content-disposition"` - ContentType string `json:"protected-content-type,omitempty"` - ErrorMessage string `json:"error,omitempty"` + ContentID string `json:"content-id"` + ContentKey []byte `json:"content-encryption-key"` + StorageMode int `json:"storage-mode"` + Output string `json:"protected-content-location"` + FileName string `json:"protected-content-disposition"` + Size int64 `json:"protected-content-length"` + Checksum string `json:"protected-content-sha256"` + ContentType string `json:"protected-content-type,omitempty"` } +const ( + Storage_none = 0 + Storage_s3 = 1 + Storage_fs = 2 +) + func writeRequestFileToTemp(r io.Reader) (int64, *os.File, error) { dir := os.TempDir() file, err := ioutil.TempFile(dir, "readium-lcp") @@ -69,9 +75,10 @@ func cleanupTempFile(f *os.File) { os.Remove(f.Name()) } -// StoreContent stores content in the storage. +// StoreContent stores content passed through the request body into the storage. // The content name is given in the url (name) -// A temporary file is created, then deleted after the content has been stored +// A temporary file is created, then deleted after the content has been stored. +// This function is using an async task. func StoreContent(w http.ResponseWriter, r *http.Request, s Server) { vars := mux.Vars(r) @@ -121,40 +128,42 @@ func AddContent(w http.ResponseWriter, r *http.Request, s Server) { problem.Error(w, r, problem.Problem{Detail: "The content id must be set in the url"}, http.StatusBadRequest) return } - // open the encrypted file, use its full path - file, err := getAndOpenFile(publication.Output) - if err != nil { - problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) - return - } - // the input file will be deleted when the function returns - defer cleanupTempFile(file) - // add the file to the storage, named by contentID, without file extension - _, err = s.Store().Add(contentID, file) - if err != nil { - problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) - return + // if the encrypted publication has not been stored yet + if publication.StorageMode == Storage_none { + + // open the encrypted file, use its full path + file, err := getAndOpenFile(publication.Output) + if err != nil { + problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) + return + } + // the input file will be deleted when the function returns + defer cleanupTempFile(file) + + // add the file to the storage, named by contentID, without file extension + _, err = s.Store().Add(contentID, file) + if err != nil { + problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) + return + } } // insert a row in the database if the content id does not already exist - // udpate the database with a new content key and file location if the content id already exists + // or update the database with a new content key and file location if the content id already exists var c index.Content c, err = s.Index().Get(contentID) - // set the encryption key (c.EncryptionKey) c.EncryptionKey = publication.ContentKey - // set the encrypted file name (c.Location) - if publication.ContentDisposition != nil { - c.Location = *publication.ContentDisposition - c.Length = *publication.Size - c.Sha256 = *publication.Checksum - c.Type = publication.ContentType + // the Location field contains either the file name (useful during download) + // or the storage URL of the publication, depending the storage mode. + if publication.StorageMode != Storage_none { + c.Location = publication.Output } else { - problem.Error(w, r, problem.Problem{Detail: "The file name must be set by the caller"}, http.StatusBadRequest) - return + c.Location = publication.FileName } - - //todo check hash & length? + c.Length = publication.Size + c.Sha256 = publication.Checksum + c.Type = publication.ContentType code := http.StatusCreated if err == index.ErrNotFound { //insert into database @@ -171,8 +180,6 @@ func AddContent(w http.ResponseWriter, r *http.Request, s Server) { // set the response http code w.WriteHeader(code) - return - } // ListContents lists the content in the storage index @@ -211,6 +218,7 @@ func GetContent(w http.ResponseWriter, r *http.Request, s Server) { } return } + // check the existence of the file item, err := s.Store().Get(contentID) if err != nil { //item probably not found @@ -223,6 +231,11 @@ func GetContent(w http.ResponseWriter, r *http.Request, s Server) { } // opens the file contentReadCloser, err := item.Contents() + if err != nil { + problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) + return + } + defer contentReadCloser.Close() if err != nil { //file probably not found problem.Error(w, r, problem.Problem{Detail: err.Error()}, http.StatusBadRequest) @@ -235,20 +248,17 @@ func GetContent(w http.ResponseWriter, r *http.Request, s Server) { // returns the content of the file to the caller io.Copy(w, contentReadCloser) - - return - } // getAndOpenFile opens a file from a path, or downloads then opens it if its location is a URL func getAndOpenFile(filePathOrURL string) (*os.File, error) { - HTTPOrHTTPS, err := isHTTPOrHTTPS(filePathOrURL) + isURL, err := isURL(filePathOrURL) if err != nil { return nil, err } - if HTTPOrHTTPS { + if isURL { return downloadAndOpenFile(filePathOrURL) } @@ -268,12 +278,11 @@ func downloadAndOpenFile(url string) (*os.File, error) { return os.Open(fileName) } -func isHTTPOrHTTPS(filePathOrURL string) (bool, error) { +func isURL(filePathOrURL string) (bool, error) { url, err := url.Parse(filePathOrURL) if err != nil { - return false, errors.New("Error parsing input file") + return false, errors.New("error parsing input string") } - return url.Scheme == "http" || url.Scheme == "https", nil } diff --git a/lcpserver/lcpserver.go b/lcpserver/lcpserver.go index e7b182fb..9c96de6b 100644 --- a/lcpserver/lcpserver.go +++ b/lcpserver/lcpserver.go @@ -76,9 +76,6 @@ func main() { if dbURI = config.Config.LcpServer.Database; dbURI == "" { dbURI = "sqlite3://file:lcp.sqlite?cache=shared&mode=rwc" } - if storagePath = config.Config.Storage.FileSystem.Directory; storagePath == "" { - storagePath = "files" - } if certFile = config.Config.Certificate.Cert; certFile == "" { panic("Must specify a certificate") } @@ -111,16 +108,22 @@ func main() { panic(err) } - // move config - license.CreateDefaultLinks() - var store storage.Store + err = license.CreateDefaultLinks() + if err != nil { + panic(err) + } + var store storage.Store if mode := config.Config.Storage.Mode; mode == "s3" { s3Conf := s3ConfigFromYAML() store, _ = storage.S3(s3Conf) - } else { + } else if config.Config.Storage.FileSystem.Directory != "" { + storagePath = config.Config.Storage.FileSystem.Directory os.MkdirAll(storagePath, os.ModePerm) //ignore the error, the folder can already exist - store = storage.NewFileSystem(storagePath, config.Config.LcpServer.PublicBaseUrl+"/files") + store = storage.NewFileSystem(storagePath, config.Config.Storage.FileSystem.URL) + log.Println("Storage created, path", storagePath, ", URL", config.Config.Storage.FileSystem.URL) + } else { + log.Println("No storage created") } packager := pack.NewPackager(store, idx, 4) @@ -146,10 +149,6 @@ func main() { } log.Println("Using database " + dbURI) log.Println("Public base URL=" + config.Config.LcpServer.PublicBaseUrl) - log.Println("License links:") - for nameOfLink, link := range config.Config.License.Links { - log.Println(" " + nameOfLink + " => " + link) - } if err := s.ListenAndServe(); err != nil { log.Println("Error " + err.Error()) diff --git a/license/license.go b/license/license.go index 7c375c7e..2611aa5e 100644 --- a/license/license.go +++ b/license/license.go @@ -9,8 +9,10 @@ import ( "crypto/rand" "crypto/tls" "encoding/base64" + "errors" "fmt" "io" + "net/url" "reflect" "strings" "time" @@ -162,15 +164,30 @@ func Initialize(contentID string, l *License) { } // CreateDefaultLinks inits the global var DefaultLinks from config data -func CreateDefaultLinks() { +func CreateDefaultLinks() error { configLinks := config.Config.License.Links + // the storage url should now be in the storage section. + storageURL := config.Config.Storage.FileSystem.URL DefaultLinks = make(map[string]string) for key := range configLinks { DefaultLinks[key] = configLinks[key] } + // this value supercedes a (deprecated) publication link placed in the license section; + // keep backward compatibility. + if storageURL != "" { + u, err := url.Parse(storageURL) + if err != nil { + return err + } + if !strings.HasSuffix(u.Path, "/") { + u.Path = u.Path + "/" + } + DefaultLinks["publication"] = u.String() + "{publication_id}" + } + return nil } // setDefaultLinks sets a Link array from config links @@ -211,28 +228,54 @@ func SetLicenseLinks(l *License, c index.Content) error { // append default links to custom links l.Links = appendDefaultLinks(&l.Links) + // check if the publication link is in the content database + hasPubLink, err := isURL(c.Location) + if err != nil { + return err + } + for i := 0; i < len(l.Links); i++ { - // set publication link + // set the publication link if l.Links[i].Rel == "publication" { - l.Links[i].Href = strings.Replace(l.Links[i].Href, "{publication_id}", l.ContentID, 1) + if hasPubLink { + // this happens only in case the configuration is broken + l.Links[i].Href = c.Location + l.Links[i].Title = l.ContentID + hasPubLink = false + } else { + l.Links[i].Href = strings.Replace(l.Links[i].Href, "{publication_id}", l.ContentID, 1) + l.Links[i].Title = c.Location + } l.Links[i].Type = c.Type l.Links[i].Size = c.Length - l.Links[i].Title = c.Location l.Links[i].Checksum = c.Sha256 } - // set status link + // set the status link if l.Links[i].Rel == "status" { l.Links[i].Href = strings.Replace(l.Links[i].Href, "{license_id}", l.ID, 1) l.Links[i].Type = api.ContentType_LSD_JSON } - // set hint page link + // set the hint page link, which may be associated with a specific license if l.Links[i].Rel == "hint" { l.Links[i].Href = strings.Replace(l.Links[i].Href, "{license_id}", l.ID, 1) l.Links[i].Type = api.ContentType_TEXT_HTML } } + // add the publication link present in the content index + if hasPubLink { + link := Link{ + Rel: "publication", + Href: c.Location, + Title: l.ContentID, + Type: c.Type, + Size: c.Length, + Checksum: c.Sha256, + } + l.Links = append(l.Links, link) + } + return nil } @@ -319,3 +362,11 @@ func SignLicense(l *License, cert *tls.Certificate) error { return nil } + +func isURL(filePathOrURL string) (bool, error) { + url, err := url.Parse(filePathOrURL) + if err != nil { + return false, errors.New("error parsing the input string") + } + return url.Scheme == "http" || url.Scheme == "https", nil +} diff --git a/pack/pack.go b/pack/pack.go index c9b1933c..461b8509 100644 --- a/pack/pack.go +++ b/pack/pack.go @@ -14,7 +14,6 @@ import ( "github.com/readium/readium-lcp-server/crypto" "github.com/readium/readium-lcp-server/epub" - "github.com/readium/readium-lcp-server/license" "github.com/readium/readium-lcp-server/xmlenc" ) @@ -27,7 +26,7 @@ type PackageReader interface { // PackageWriter is an interface type PackageWriter interface { NewFile(path string, contentType string, storageMethod uint16) (io.WriteCloser, error) - MarkAsEncrypted(path string, originalSize int64, profile license.EncryptionProfile, algorithm string) + MarkAsEncrypted(path string, originalSize int64, algorithm string) Close() error } @@ -44,7 +43,7 @@ type Resource interface { } // Process copies resources from the source to the destination package, after encryption if needed. -func Process(profile license.EncryptionProfile, encrypter crypto.Encrypter, reader PackageReader, writer PackageWriter) (key crypto.ContentKey, err error) { +func Process(encrypter crypto.Encrypter, reader PackageReader, writer PackageWriter) (key crypto.ContentKey, err error) { // generate an encryption key key, err = encrypter.GenerateKey() @@ -63,7 +62,7 @@ func Process(profile license.EncryptionProfile, encrypter crypto.Encrypter, read // loop through the resources of the source package, encrypt them if needed, copy them into the dest package for _, resource := range reader.Resources() { if !resource.Encrypted() && resource.CanBeEncrypted() { - err = encryptRPFResource(compressor, profile, encrypter, key, resource, writer) + err = encryptRPFResource(compressor, encrypter, key, resource, writer) if err != nil { log.Println("Error encrypting " + resource.Path() + ": " + err.Error()) return @@ -166,7 +165,7 @@ func canEncrypt(file *epub.Resource, ep epub.Epub) bool { } // encryptRPFResource encrypts a resource in a Readium Package -func encryptRPFResource(compressor *flate.Writer, profile license.EncryptionProfile, encrypter crypto.Encrypter, key crypto.ContentKey, resource Resource, packageWriter PackageWriter) error { +func encryptRPFResource(compressor *flate.Writer, encrypter crypto.Encrypter, key crypto.ContentKey, resource Resource, packageWriter PackageWriter) error { // add the file to the package writer // note: the file is stored as-is because compression, when applied, is applied *before* encryption @@ -199,7 +198,7 @@ func encryptRPFResource(compressor *flate.Writer, profile license.EncryptionProf resourceReader.Close() file.Close() - packageWriter.MarkAsEncrypted(resource.Path(), resource.Size(), profile, encrypter.Signature()) + packageWriter.MarkAsEncrypted(resource.Path(), resource.Size(), encrypter.Signature()) return err } diff --git a/pack/rwppackage.go b/pack/rwppackage.go index 83bdf190..5d445bec 100644 --- a/pack/rwppackage.go +++ b/pack/rwppackage.go @@ -12,7 +12,6 @@ import ( "os" "text/template" - "github.com/readium/readium-lcp-server/license" "github.com/readium/readium-lcp-server/rwpm" ) @@ -200,10 +199,10 @@ func (writer *RPFWriter) NewFile(path string, contentType string, storageMethod return &NopWriteCloser{w}, err } -// MarkAsEncrypted marks a resource as encrypted (with an lcp profile and algorithm), in the writer manifest +// MarkAsEncrypted marks a resource as encrypted (with an algorithm), in the writer manifest // FIXME: currently only looks into the reading order. Add "alternates", think about adding "resources" // FIXME: process resources which are compressed before encryption -> add Compression and OriginalLength properties in this case -func (writer *RPFWriter) MarkAsEncrypted(path string, originalSize int64, profile license.EncryptionProfile, algorithm string) { +func (writer *RPFWriter) MarkAsEncrypted(path string, originalSize int64, algorithm string) { for i, resource := range writer.manifest.ReadingOrder { if path == resource.Href { @@ -293,7 +292,7 @@ func OpenRPF(name string) (*RPFReader, error) { // BuildRPFFromPDF builds a Readium Package (rwpp) which embeds a PDF file func BuildRPFFromPDF(title string, inputPath string, outputPath string) error { - // crate the rwpp + // create the rwpp f, err := os.Create(outputPath) if err != nil { return err diff --git a/pack/rwppackage_test.go b/pack/rwppackage_test.go index 5b4ff3fa..14c12792 100644 --- a/pack/rwppackage_test.go +++ b/pack/rwppackage_test.go @@ -11,7 +11,6 @@ import ( "testing" "time" - "github.com/readium/readium-lcp-server/license" "github.com/readium/readium-lcp-server/rwpm" ) @@ -59,7 +58,7 @@ func TestWriteRPFackage(t *testing.T) { t.Fatalf("Could not close file, %s", err) } - writer.MarkAsEncrypted("test.pdf", 4, license.BasicProfile, "http://www.w3.org/2001/04/xmlenc#aes256-cbc") + writer.MarkAsEncrypted("test.pdf", 4, "http://www.w3.org/2001/04/xmlenc#aes256-cbc") err = writer.Close() if err != nil {