diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cf4ddfd --- /dev/null +++ b/README.md @@ -0,0 +1,147 @@ +#About +A file-like [MogileFS](http://danga.com/mogilefs/ "Danga Interactive - MogileFS") client for Java. + +#Features +* `java.io.File` like API +* Supports streams of unknown length +* Unit/Integration tests +* Spring friendly +* Tracker connection pooling with balancing between hosts and strategies for dealing with failed nodes +* Local file system implementation for faking in tests (`fm.last.moji.local.LocalFileSystemMoji`) + +#Configuration +### Using plain-old-Java + String hosts = "192.168.0.1,192.168.0.2"; + String domain = "testdomain"; + + Moji moji = new SpringMojiBean(hosts, domain); + moji.setTestOnBorrow(true); +### Using the Spring framework +Set some properties for your context: + + moji.tracker.address=192.168.0.1,192.168.0.2 + moji.domain=testdomain + +Import the Moji Spring context: + + + +*Or* create a Moji spring bean: + + + + + + + + + +#Usage +####Create/update a remote file + MojiFile rickRoll = moji.getFile("rick-astley"); + moji.copyToMogile(new File("never-gonna-give-you-up.mp3"), rickRoll); + +Or in a given storage class: + + MojiFile rickRoll = moji.getFile("rick-astley", "music-meme"); + +####Get the remote file size + long length = rickRoll.length(); +####Rename the remote file + rickRoll.rename("stairway-to-heaven"); +####Check the existence of a remote file + MojiFile abba = moji.getFile("voulez-vous"); + if (abba.exists()) { + ... +####Delete the remote file + abba.delete(); +####Download a remote file + MojiFile fooFighters = moji.getFile("stacked-actors"); + fooFighters.copyToFile(new File("foo-fighters.mp3")); +####Modify the storage class of a remote file + fooFighters.modifyStorageClass("awesome"); + +####Stream from a remote file + InputStream stream = null; + try { + stream = fooFighters.getInputStream(); + // Do something streamy + // stream.read(); + } finally { + stream.close(); + } + +####Stream to a remote file +This will either create a new file or overwrite an existing file's contents + + OutputStream stream = null; + try { + stream = fooFighters.getOutputStream(); + // Do something streamy + // stream.write(...); + stream.flush(); + } finally { + stream.close(); + } +####List remote files by prefix + List files = moji.list("abba-"); + for(MojiFile file : files) { + // abba-waterloo, abba-voulez-vous, abba-fernado, etc. + } + +Impose a limit on the number of items returned: + + List files = moji.list("abba-", 10); + for(MojiFile file : files) { + // abba-waterloo, abba-voulez-vous, abba-fernado, etc. - maximum of 10 + } + +####Get the locations of a remote file + File fooFighters = moji.getFile("in-your-honour"); + List paths = fooFighters.getPaths(); + // http://192.168.0.2:7500/dev2/0/000/000/0000000819.fid, http://192.168.0.4:7500/dev3/0/000/000/0000000819.fid, etc + +#Running the integration tests +To run the integration tests you need: + +* A Linux system - FIXME +* A test MogileFS tracker and a storage node ([installation instructions](http://code.google.com/p/mogilefs/wiki/InstallHowTo "Google Code - MogileFS installation instructions")) +* `mogtool, mogupload` +* `uuencode` (apt-get install sharutil) + +MogileFS integration test properties config: + +* These properties should be set in `/moji.properties` +* Set your Tracker address with the property: + + moji.tracker.hosts +* Declare your Mogile domain with the property: + + moji.domain +* Declare two storage classes in your Mogile instance and assign them with these properties: + + test.moji.class.a + test.moji.class.b +* Choose a key prefix to avoid any key clashes with real data (you're using a test instance right?) or other tests. Otherwise we might get unexpected behaviour and file deletions: + + test.moji.key.prefix + +#Building +This project uses the [Maven](http://maven.apache.org/) build system. +#Further work +We plan to support the [Java 7 FileSystem abstraction](http://openjdk.java.net/projects/nio/ "OpenJDK: NIO") now it's been officially released. + +#Legal +Copyright 2011 [Last.fm](http://www.last.fm/) + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. \ No newline at end of file diff --git a/changelog.txt b/changelog.txt new file mode 100644 index 0000000..3b24f74 --- /dev/null +++ b/changelog.txt @@ -0,0 +1,2 @@ +TBA +- Initial release \ No newline at end of file diff --git a/eclipse-codeformatter-profile.xml b/eclipse-codeformatter-profile.xml new file mode 100644 index 0000000..78b960c --- /dev/null +++ b/eclipse-codeformatter-profile.xml @@ -0,0 +1,267 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..b6334e9 --- /dev/null +++ b/pom.xml @@ -0,0 +1,255 @@ + + + 4.0.0 + + fm.last.commons + moji + 1.1.0-SNAPSHOT + Moji + A File-like MogileFS client + + https://github.com/lastfm/moji + + + scm:git:ssh:git@github.com:lastfm/moji.git + + + + + Elliot West + teabot@gmail.com + http://www.last.fm/user/teabot + + Documentation + Java Developer + + Last.fm + http://www.last.fm/ + + + James Grant + Last.fm + http://www.last.fm/ + + Documentation + Java Developer + + + + + + Last.fm + http://www.last.fm + + + + true + true + + + + + + ${project.basedir}/src/test/resources/moji.properties${moji.env} + + + + + src/main/resources + + + src/main/conf + + + src/test/script + true + + **/*.sh + + + + src/test/script + true + + **/*.sh + + + + + + + src/test/resources + + + src/test/conf + + + + + + maven-compiler-plugin + + 1.6 + 1.6 + UTF-8 + true + + + + org.apache.maven.plugins + maven-resources-plugin + + UTF-8 + + + + org.codehaus.mojo + exec-maven-plugin + 1.1 + + + annoying-chmod + pre-integration-test + + ${project.basedir}/src/test/script/permission-change.sh + + + exec + + + + init-mogile-fs-test-data + pre-integration-test + + ${project.basedir}/target/classes/init-mogile-test-data.sh + ${project.basedir}/src/test + + + exec + + + + + + maven-failsafe-plugin + 2.6 + + + + integration-test + verify + + + + + + org.apache.maven.plugins + maven-site-plugin + 3.0 + + + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + 2.7 + + + http://commons.apache.org/pool/api-1.5.6/ + + true + true + + + + org.codehaus.mojo + cobertura-maven-plugin + 2.5.1 + + + xml + html + + + + + org.codehaus.mojo + findbugs-maven-plugin + 2.3.2 + + ${project.basedir}/src/test/resources/findbugsExclude.xml + + + + org.apache.maven.plugins + maven-checkstyle-plugin + 2.6 + + ${project.basedir}/src/test/resources/checkstyle.xml + + + + + + + + hudson + + + env + hudson + + + + .hudson + + + + + + + org.slf4j + slf4j-api + 1.6.1 + + + commons-io + commons-io + 2.0.1 + + + org.apache.commons + commons-pool + 1.5.6 + + + + commons-lang + commons-lang + 2.1 + test + + + org.slf4j + slf4j-log4j12 + 1.6.1 + test + + + junit + junit + 4.8.2 + test + + + org.mockito + mockito-all + 1.8.5 + test + + + diff --git a/src/main/java/fm/last/moji/Moji.java b/src/main/java/fm/last/moji/Moji.java new file mode 100644 index 0000000..d5b03d5 --- /dev/null +++ b/src/main/java/fm/last/moji/Moji.java @@ -0,0 +1,84 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji; + +import java.io.File; +import java.io.IOException; +import java.util.List; + +/** + * The Moji entry point. A representation of a MogileFS domain that allows interactions with remote files. + *

+ * Example usage: + *

+ * + *

+ * MojiFactory factory = new PropertyMojiFactory();
+ * Moji moji = factory.getInstance();
+ * MojiFile file = moji.getFile("some-key");
+ * file.copyToFile(new File("localFile.dat"));
+ * 
+ */ +public interface Moji { + + /** + * Creates an abstract representation of a remote MogileFS file for the given key. + * + * @param key MogileFS file key. + * @return Representation of the remote file. + */ + MojiFile getFile(String key); + + /** + * Creates an abstract representation of a remote MogileFS file for the given key. When the file content is modified + * the file will also be assigned the specified storage class. Note that storage class parameter has no effect when + * reading files. + * + * @param key MogileFS file key. + * @param storageClass The storage class to which a new file will be assigned. + * @return Representation of the remote file. + */ + MojiFile getFile(String key, String storageClass); + + /** + * Copies a local source file to the given remote MogileFS destination file. + * + * @param source The local source file. + * @param destination The remote destination of the file. + * @throws IOException If there was a problem writing the file. + */ + void copyToMogile(File source, MojiFile destination) throws IOException; + + /** + * Get remote MogileFS file representations that match the given key prefix. + * + * @param keyPrefix The Key prefix to match remote files on. + * @return A list of matching MofileFS file representations or an empty list if there were no matches. + * @throws IOException If there was a problem communicating with MogileFS. + */ + List list(String keyPrefix) throws IOException; + + /** + * Get a bounded list of remote MogileFS file representations that match the given key prefix. + * + * @param keyPrefix The Key prefix to match remote files on. + * @param limit The maximum number of files to return. + * @return A list of matching MofileFS file representations or an empty list if there were no matches. + * @throws IOException If there was a problem communicating with MogileFS. + */ + List list(String keyPrefix, int limit) throws IOException; + +} diff --git a/src/main/java/fm/last/moji/MojiFactory.java b/src/main/java/fm/last/moji/MojiFactory.java new file mode 100644 index 0000000..f14f60a --- /dev/null +++ b/src/main/java/fm/last/moji/MojiFactory.java @@ -0,0 +1,41 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji; + +import java.io.IOException; + +/** + * Factory interface for creating {@link fm.last.moji.Moji Mojis}. + */ +public interface MojiFactory { + + /** + * Gets a Moji instance using whatever strategy the implemention provides. + * + * @return The Moji instance. + * @throws IOException If there was a problem creating the instance. + */ + Moji getInstance() throws IOException; + + /** + * Gets a Moji instance using whatever strategy the implemention provides. + * + * @return The Moji instance. + * @throws IOException If there was a problem creating the instance. + */ + Moji getInstance(String domain) throws IOException; + +} diff --git a/src/main/java/fm/last/moji/MojiFile.java b/src/main/java/fm/last/moji/MojiFile.java new file mode 100644 index 0000000..d2dd4e1 --- /dev/null +++ b/src/main/java/fm/last/moji/MojiFile.java @@ -0,0 +1,123 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.URL; +import java.util.List; + +/** + * An abstract representation of a remote MogileFS file. + */ +public interface MojiFile { + + /** + * Determines whether or not the file represented by this object exists in MogileFS. + * + * @return true if the file exists in MogileFS, false otherwise. + * @throws IOException If there was a problem communicating with MogileFS. + */ + boolean exists() throws IOException; + + /** + * Deletes this file in MogileFS if it exists. If the file doesn't exists then this method will return without error. + * + * @throws IOException If there was a problem communicating with MogileFS. + */ + void delete() throws IOException; + + /** + * Assigns a new key to the MogileFS file represented by this object. + * + * @param key The new key for the file. + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws fm.last.moji.tracker.KeyExistsAlreadyException If the new key already points to a file in MogileFS. + * @throws IOException If there was a problem communicating with MogileFS. + */ + void rename(String key) throws IOException; + + /** + * Gets an InputStream to the content of the MogileFS file represented by this object. + * + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws IOException If there was a problem communicating with MogileFS. + */ + InputStream getInputStream() throws IOException; + + /** + * Gets an OutputStream for writing content to the MogileFS file represented by this object. If the file does not + * exist then it will be created. + * + * @throws IOException If there was a problem communicating with MogileFS. + */ + OutputStream getOutputStream() throws IOException; + + /** + * Copies the content of the file represented by this object to a local file destination. + * + * @param file + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws IOException + */ + void copyToFile(File file) throws IOException; + + /** + * Returns the length in bytes of this remote file. + * + * @return File length in bytes. + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws IOException If there was a problem communicating with MogileFS. + */ + long length() throws IOException; + + /** + * Assigns this file to the specified storage class. + * + * @param storageClass The new storage class + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws fm.last.moji.tracker.UnknownStorageClassException If the specified storage class is not defined in + * MogileFS. + * @throws IOException If there was a problem communicating with MogileFS. + */ + void modifyStorageClass(String storageClass) throws IOException; + + /** + * Returns a list of storage node paths from which this file can be accessed. + * + * @return A list of storage node paths or an empty list. + * @throws fm.last.moji.tracker.UnknownKeyException If this file doesn't exist in MogileFS. + * @throws IOException If there was a problem communicating with MogileFS. + */ + List getPaths() throws IOException; + + /** + * The key of this file in MogileFS. + * + * @return This files key. + */ + String getKey(); + + /** + * The MogileFS domain in which this file is located. + * + * @return This files key. + */ + String getDomain(); + +} diff --git a/src/main/java/fm/last/moji/impl/DefaultMojiFactory.java b/src/main/java/fm/last/moji/impl/DefaultMojiFactory.java new file mode 100644 index 0000000..13808d6 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/DefaultMojiFactory.java @@ -0,0 +1,49 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFactory; +import fm.last.moji.tracker.TrackerFactory; + +/** + * Creates a {@link fm.last.moji.Moji Moji} instance. + */ +public class DefaultMojiFactory implements MojiFactory { + + private final String defaultDomain; + private final TrackerFactory trackerFactory; + private final HttpConnectionFactory httpFactory; + + public DefaultMojiFactory(TrackerFactory trackerFactory, String defaultDomain) { + this.trackerFactory = trackerFactory; + this.defaultDomain = defaultDomain; + httpFactory = new HttpConnectionFactory(trackerFactory.getProxy()); + } + + @Override + public Moji getInstance() { + return new MojiImpl(trackerFactory, httpFactory, defaultDomain); + } + + @Override + public Moji getInstance(String domain) throws IOException { + return new MojiImpl(trackerFactory, httpFactory, domain); + } + +} diff --git a/src/main/java/fm/last/moji/impl/DeleteCommand.java b/src/main/java/fm/last/moji/impl/DeleteCommand.java new file mode 100644 index 0000000..1ebb48e --- /dev/null +++ b/src/main/java/fm/last/moji/impl/DeleteCommand.java @@ -0,0 +1,48 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import fm.last.moji.tracker.Tracker; + +class DeleteCommand implements MojiCommand { + + final String key; + final String domain; + + DeleteCommand(String key, String domain) { + this.key = key; + this.domain = domain; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + tracker.delete(key, domain); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("DeleteCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/Executor.java b/src/main/java/fm/last/moji/impl/Executor.java new file mode 100644 index 0000000..ad65705 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/Executor.java @@ -0,0 +1,69 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerFactory; +import fm.last.moji.tracker.impl.CommunicationException; + +class Executor { + + private static final Logger log = LoggerFactory.getLogger(Executor.class); + + private final TrackerFactory trackerFactory; + private int maxAttempts; + + Executor(TrackerFactory trackerFactory) { + this.trackerFactory = trackerFactory; + maxAttempts = trackerFactory.getAddresses().size(); + } + + public void executeCommand(MojiCommand command) throws IOException { + Tracker tracker = null; + CommunicationException lastException = null; + for (int attempt = 0; attempt < maxAttempts; attempt++) { + try { + tracker = trackerFactory.getTracker(); + log.debug("executing {}", command); + if (maxAttempts > 1) { + log.debug("Attempt #{}", attempt); + } + command.executeWithTracker(tracker); + return; + } catch (CommunicationException e) { + lastException = e; + } finally { + if (tracker != null) { + tracker.close(); + } + } + } + if (maxAttempts > 1) { + log.debug("All {} attempts failed", maxAttempts); + } + throw lastException; + } + + void setMaxAttempts(int maxAttempts) { + this.maxAttempts = maxAttempts; + } + +} diff --git a/src/main/java/fm/last/moji/impl/ExistsCommand.java b/src/main/java/fm/last/moji/impl/ExistsCommand.java new file mode 100644 index 0000000..be8752c --- /dev/null +++ b/src/main/java/fm/last/moji/impl/ExistsCommand.java @@ -0,0 +1,63 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.net.URL; +import java.util.List; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.UnknownKeyException; + +class ExistsCommand implements MojiCommand { + + final String key; + final String domain; + private boolean exists; + + ExistsCommand(String key, String domain) { + this.key = key; + this.domain = domain; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + List paths = null; + try { + paths = tracker.getPaths(key, domain); + exists = !paths.isEmpty(); + } catch (UnknownKeyException e) { + } + } + + boolean getExists() { + return exists; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("ExistsCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", exists="); + builder.append(exists); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/FileDownloadInputStream.java b/src/main/java/fm/last/moji/impl/FileDownloadInputStream.java new file mode 100644 index 0000000..b07a922 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/FileDownloadInputStream.java @@ -0,0 +1,95 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.io.InputStream; +import java.util.concurrent.locks.Lock; + +import org.apache.commons.io.input.CountingInputStream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +class FileDownloadInputStream extends InputStream { + + private static final Logger log = LoggerFactory.getLogger(FileDownloadInputStream.class); + + private final CountingInputStream delegate; + private final Lock readLock; + + FileDownloadInputStream(InputStream delegate, Lock readLock) { + this.readLock = readLock; + this.delegate = new CountingInputStream(delegate); + } + + @Override + public int read() throws IOException { + return delegate.read(); + } + + @Override + public int read(byte[] b) throws IOException { + return delegate.read(b); + } + + @Override + public int read(byte[] b, int off, int len) throws IOException { + return delegate.read(b, off, len); + } + + @Override + public long skip(long n) throws IOException { + return delegate.skip(n); + } + + @Override + public int available() throws IOException { + return delegate.available(); + } + + @Override + public void close() throws IOException { + log.debug("Read {} bytes", delegate.getCount()); + try { + delegate.close(); + } finally { + unlockQuietly(readLock); + } + } + + @Override + public void mark(int readlimit) { + delegate.mark(readlimit); + } + + @Override + public void reset() throws IOException { + delegate.reset(); + } + + @Override + public boolean markSupported() { + return delegate.markSupported(); + } + + private void unlockQuietly(Lock lock) { + try { + lock.unlock(); + } catch (IllegalMonitorStateException e) { + } + } + +} diff --git a/src/main/java/fm/last/moji/impl/FileLengthCommand.java b/src/main/java/fm/last/moji/impl/FileLengthCommand.java new file mode 100644 index 0000000..a9c3fe6 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/FileLengthCommand.java @@ -0,0 +1,100 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.HttpURLConnection; +import java.net.URL; +import java.util.List; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Tracker; + +class FileLengthCommand implements MojiCommand { + + private static final Logger log = LoggerFactory.getLogger(FileLengthCommand.class); + + private final HttpConnectionFactory httpFactory; + final String key; + final String domain; + private long length = -1L; + + FileLengthCommand(HttpConnectionFactory httpFactory, String key, String domain) { + this.httpFactory = httpFactory; + this.key = key; + this.domain = domain; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + List paths = tracker.getPaths(key, domain); + if (!paths.isEmpty()) { + HttpURLConnection httpConnection = null; + IOException lastException = null; + for (URL path : paths) { + try { + log.debug("HTTP HEAD -> {}", path); + httpConnection = httpFactory.newConnection(path); + httpConnection.setRequestMethod("HEAD"); + length = httpConnection.getContentLength(); + log.debug("Content-Length: {}", length); + return; + } catch (IOException e) { + log.debug("Failed to open input -> {}", path); + log.debug("Exception was: ", e); + lastException = e; + } finally { + if (httpConnection != null) { + httpConnection.disconnect(); + } + } + } + throw lastException; + } else { + log.debug("No paths found for domain={},key={} - throwing", domain, key); + throw new FileNotFoundException("domain=" + domain + ",key=" + key); + } + } + + long getLength() { + return length; + } + + String getKey() { + return key; + } + + String getDomain() { + return domain; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("FileLengthCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", length="); + builder.append(length); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/FileUploadOutputStream.java b/src/main/java/fm/last/moji/impl/FileUploadOutputStream.java new file mode 100644 index 0000000..25d7e5f --- /dev/null +++ b/src/main/java/fm/last/moji/impl/FileUploadOutputStream.java @@ -0,0 +1,140 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.io.OutputStream; +import java.net.HttpURLConnection; +import java.util.concurrent.locks.Lock; + +import org.apache.commons.io.output.CountingOutputStream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerFactory; + +class FileUploadOutputStream extends OutputStream { + + private static final Logger log = LoggerFactory.getLogger(FileUploadOutputStream.class); + + private static final int CHUNK_LENGTH = 4096; + private final Destination destination; + private final TrackerFactory trackerFactory; + private final String key; + private final String domain; + private final Lock writeLock; + private final HttpURLConnection httpConnection; + private final CountingOutputStream delegate; + + FileUploadOutputStream(TrackerFactory trackerFactory, HttpConnectionFactory httpFactory, String key, String domain, + Destination destination, Lock writeLock) throws IOException { + this.destination = destination; + this.trackerFactory = trackerFactory; + this.domain = domain; + this.key = key; + this.writeLock = writeLock; + + log.debug("HTTP PUT -> opening chunked stream -> {}", destination.getPath()); + httpConnection = httpFactory.newConnection(destination.getPath()); + httpConnection.setRequestMethod("PUT"); + httpConnection.setChunkedStreamingMode(CHUNK_LENGTH); + httpConnection.setDoOutput(true); + delegate = new CountingOutputStream(httpConnection.getOutputStream()); + } + + @Override + public void write(int b) throws IOException { + delegate.write(b); + } + + @Override + public void write(byte[] b) throws IOException { + delegate.write(b); + } + + @Override + public void write(byte[] b, int off, int len) throws IOException { + delegate.write(b, off, len); + } + + @Override + public void flush() throws IOException { + delegate.flush(); + } + + @Override + public void close() throws IOException { + log.debug("Close called on {}", this); + long size = -1L; + try { + try { + delegate.flush(); + size = delegate.getByteCount(); + } finally { + try { + delegate.close(); + } finally { + try { + String message = httpConnection.getResponseMessage(); + int code = httpConnection.getResponseCode(); + if (HttpURLConnection.HTTP_OK != code) { + throw new IOException(code + " " + message); + } else { + log.debug("Status: HTTP {} - {}", code, message); + } + } finally { + httpConnection.disconnect(); + } + } + } + } finally { + log.debug("Bytes written: {}", size); + Tracker tracker = trackerFactory.getTracker(); + try { + tracker.createClose(key, domain, destination, size); + } finally { + try { + tracker.close(); + } finally { + unlockQuietly(writeLock); + } + } + } + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("FileUploadOutputStream [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", destination="); + builder.append(destination); + builder.append("]"); + return builder.toString(); + } + + private void unlockQuietly(Lock lock) { + try { + lock.unlock(); + } catch (IllegalMonitorStateException e) { + } + } + +} diff --git a/src/main/java/fm/last/moji/impl/GetInputStreamCommand.java b/src/main/java/fm/last/moji/impl/GetInputStreamCommand.java new file mode 100644 index 0000000..5167fff --- /dev/null +++ b/src/main/java/fm/last/moji/impl/GetInputStreamCommand.java @@ -0,0 +1,87 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.net.HttpURLConnection; +import java.net.URL; +import java.util.List; +import java.util.concurrent.locks.Lock; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Tracker; + +class GetInputStreamCommand implements MojiCommand { + + private static final Logger log = LoggerFactory.getLogger(GetInputStreamCommand.class); + + final String key; + final String domain; + private final HttpConnectionFactory httpFactory; + private final Lock readLock; + private InputStream stream; + + GetInputStreamCommand(String key, String domain, HttpConnectionFactory httpFactory, Lock readLock) { + this.key = key; + this.domain = domain; + this.httpFactory = httpFactory; + this.readLock = readLock; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + List paths = tracker.getPaths(key, domain); + if (paths.isEmpty()) { + throw new FileNotFoundException("key=" + key + ", domain=" + domain); + } + IOException lastException = null; + for (URL path : paths) { + try { + log.debug("Opened: {}", path); + HttpURLConnection urlConnection = httpFactory.newConnection(path); + stream = new FileDownloadInputStream(urlConnection.getInputStream(), readLock); + return; + } catch (IOException e) { + log.debug("Failed to open input -> {}", path); + log.debug("Exception was: ", e); + lastException = e; + IOUtils.closeQuietly(stream); + } + } + throw lastException; + } + + InputStream getInputStream() { + return stream; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("GetInputStreamCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/GetOutputStreamCommand.java b/src/main/java/fm/last/moji/impl/GetOutputStreamCommand.java new file mode 100644 index 0000000..aef7b91 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/GetOutputStreamCommand.java @@ -0,0 +1,94 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.io.OutputStream; +import java.util.List; +import java.util.concurrent.locks.Lock; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.TrackerFactory; + +class GetOutputStreamCommand implements MojiCommand { + + private static final Logger log = LoggerFactory.getLogger(GetOutputStreamCommand.class); + + final String key; + final String domain; + final String storageClass; + private final TrackerFactory trackerFactory; + private final HttpConnectionFactory httpFactory; + private OutputStream stream; + private final Lock writeLock; + + GetOutputStreamCommand(TrackerFactory trackerFactory, HttpConnectionFactory httpFactory, String key, String domain, + String storageClass, Lock writeLock) { + this.trackerFactory = trackerFactory; + this.httpFactory = httpFactory; + this.key = key; + this.domain = domain; + this.storageClass = storageClass; + this.writeLock = writeLock; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + List destinations = tracker.createOpen(key, domain, storageClass); + if (destinations.isEmpty()) { + throw new TrackerException("Failed to obtain destinations for domain=" + domain + ",key=" + key + + ",storageClass=" + storageClass); + } + IOException lastException = null; + for (Destination destination : destinations) { + log.debug("Creating output stream to: {}", destination); + try { + stream = new FileUploadOutputStream(trackerFactory, httpFactory, key, domain, destinations.get(0), writeLock); + return; + } catch (IOException e) { + log.debug("Failed to open output -> {}", destination); + log.debug("Exception was: ", e); + lastException = e; + IOUtils.closeQuietly(stream); + } + } + throw lastException; + } + + OutputStream getOutputStream() { + return stream; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("GetOutputStreamCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", storageClass="); + builder.append(storageClass); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/GetPathsCommand.java b/src/main/java/fm/last/moji/impl/GetPathsCommand.java new file mode 100644 index 0000000..f0d752e --- /dev/null +++ b/src/main/java/fm/last/moji/impl/GetPathsCommand.java @@ -0,0 +1,57 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.net.URL; +import java.util.Collections; +import java.util.List; + +import fm.last.moji.tracker.Tracker; + +class GetPathsCommand implements MojiCommand { + + private final String key; + private final String domain; + private List paths; + + GetPathsCommand(String key, String domain) { + this.key = key; + this.domain = domain; + paths = Collections.emptyList(); + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + paths = tracker.getPaths(key, domain); + } + + List getPaths() { + return paths; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("GetPathsCommand [key="); + builder.append(key); + builder.append(", domain="); + builder.append(domain); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/HttpConnectionFactory.java b/src/main/java/fm/last/moji/impl/HttpConnectionFactory.java new file mode 100644 index 0000000..408c7db --- /dev/null +++ b/src/main/java/fm/last/moji/impl/HttpConnectionFactory.java @@ -0,0 +1,35 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.net.HttpURLConnection; +import java.net.Proxy; +import java.net.URL; + +class HttpConnectionFactory { + + private final Proxy proxy; + + HttpConnectionFactory(Proxy proxy) { + this.proxy = proxy; + } + + HttpURLConnection newConnection(URL url) throws IOException { + return (HttpURLConnection) url.openConnection(proxy); + } + +} diff --git a/src/main/java/fm/last/moji/impl/ListFilesCommand.java b/src/main/java/fm/last/moji/impl/ListFilesCommand.java new file mode 100644 index 0000000..ba51cf7 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/ListFilesCommand.java @@ -0,0 +1,67 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFile; +import fm.last.moji.tracker.Tracker; + +class ListFilesCommand implements MojiCommand { + + final String keyPrefix; + final String domain; + final Integer limit; + private List files; + private final Moji moji; + + ListFilesCommand(Moji moji, String keyPrefix, String domain, int limit) { + this(moji, keyPrefix, domain, Integer.valueOf(limit)); + } + + ListFilesCommand(Moji moji, String keyPrefix, String domain) { + this(moji, keyPrefix, domain, null); + } + + private ListFilesCommand(Moji moji, String keyPrefix, String domain, Integer limit) { + this.moji = moji; + this.keyPrefix = keyPrefix; + this.domain = domain; + this.limit = limit; + files = Collections.emptyList(); + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + List keys = tracker.list(domain, keyPrefix, limit); + if (!keys.isEmpty()) { + files = new ArrayList(keys.size()); + for (String key : keys) { + MojiFile file = moji.getFile(key); + files.add(file); + } + } + } + + List getFileList() { + return files; + } + +} diff --git a/src/main/java/fm/last/moji/impl/MojiCommand.java b/src/main/java/fm/last/moji/impl/MojiCommand.java new file mode 100644 index 0000000..8efeee4 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/MojiCommand.java @@ -0,0 +1,26 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import fm.last.moji.tracker.Tracker; + +interface MojiCommand { + + void executeWithTracker(Tracker tracker) throws IOException; + +} diff --git a/src/main/java/fm/last/moji/impl/MojiFileImpl.java b/src/main/java/fm/last/moji/impl/MojiFileImpl.java new file mode 100644 index 0000000..1887023 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/MojiFileImpl.java @@ -0,0 +1,241 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.URL; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.MojiFile; +import fm.last.moji.tracker.TrackerFactory; + +class MojiFileImpl implements MojiFile { + + private static final Logger log = LoggerFactory.getLogger(MojiFileImpl.class); + + private final String domain; + private final TrackerFactory trackerFactory; + private final HttpConnectionFactory httpFactory; + private final ReadWriteLock lock; + private Executor executor; + private String storageClass; + private String key; + + MojiFileImpl(String key, String domain, String storageClass, TrackerFactory trackerFactory, + HttpConnectionFactory httpFactory) { + this.key = key; + this.domain = domain; + this.storageClass = storageClass; + this.trackerFactory = trackerFactory; + this.httpFactory = httpFactory; + executor = new Executor(trackerFactory); + lock = new ReentrantReadWriteLock(); + } + + @Override + public boolean exists() throws IOException { + log.debug("exists() : {}", this); + boolean exists = false; + try { + lock.readLock().lock(); + ExistsCommand command = new ExistsCommand(key, domain); + executor.executeCommand(command); + exists = command.getExists(); + log.debug("exists() -> {}", exists); + } finally { + lock.readLock().unlock(); + } + return exists; + } + + @Override + public void delete() throws IOException { + log.debug("delete() : {}", this); + try { + lock.writeLock().lock(); + DeleteCommand command = new DeleteCommand(key, domain); + executor.executeCommand(command); + } finally { + lock.writeLock().unlock(); + } + } + + @Override + public long length() throws IOException { + log.debug("length() : {}", this); + long length = -1L; + try { + lock.readLock().lock(); + FileLengthCommand command = new FileLengthCommand(httpFactory, key, domain); + executor.executeCommand(command); + length = command.getLength(); + log.debug("length() -> {}", length); + } finally { + lock.readLock().unlock(); + } + return length; + } + + @Override + public InputStream getInputStream() throws IOException { + log.debug("getInputStream() : {}", this); + InputStream inputStream = null; + try { + Lock readLock = lock.readLock(); + readLock.lock(); + GetInputStreamCommand command = new GetInputStreamCommand(key, domain, httpFactory, readLock); + executor.executeCommand(command); + inputStream = command.getInputStream(); + log.debug("getInputStream() -> {}", inputStream); + } catch (Throwable e) { + unlockQuietly(lock.readLock()); + IOUtils.closeQuietly(inputStream); + if (e instanceof IOException) { + throw (IOException) e; + } else { + throw new RuntimeException(e); + } + } + // calling close will release the lock + return inputStream; + } + + @Override + public OutputStream getOutputStream() throws IOException { + log.debug("getOutputStream() : {}", this); + OutputStream outputStream = null; + try { + Lock writeLock = lock.writeLock(); + writeLock.lock(); + GetOutputStreamCommand command = new GetOutputStreamCommand(trackerFactory, httpFactory, key, domain, + storageClass, writeLock); + executor.executeCommand(command); + outputStream = command.getOutputStream(); + log.debug("getOutputStream() -> {}", outputStream); + } catch (Throwable e) { + unlockQuietly(lock.writeLock()); + IOUtils.closeQuietly(outputStream); + if (e instanceof IOException) { + throw (IOException) e; + } else { + throw new RuntimeException(e); + } + } + // calling close will release the lock + return outputStream; + } + + @Override + public void rename(String newKey) throws IOException { + log.debug("rename() : {}", this); + try { + lock.writeLock().lock(); + RenameCommand command = new RenameCommand(key, domain, newKey); + executor.executeCommand(command); + key = newKey; + } finally { + lock.writeLock().unlock(); + } + } + + @Override + public void modifyStorageClass(String newStorageClass) throws IOException { + log.debug("setStorageClass() : {}", this); + try { + lock.writeLock().lock(); + UpdateStorageClassCommand command = new UpdateStorageClassCommand(key, domain, newStorageClass); + executor.executeCommand(command); + storageClass = newStorageClass; + } finally { + lock.writeLock().unlock(); + } + } + + @Override + public List getPaths() throws IOException { + log.debug("getPaths() : {}", this); + List paths = Collections.emptyList(); + try { + lock.readLock().lock(); + GetPathsCommand command = new GetPathsCommand(key, domain); + executor.executeCommand(command); + paths = command.getPaths(); + log.debug("getPaths() -> {}", paths); + } finally { + lock.readLock().unlock(); + } + return paths; + } + + @Override + public String getKey() { + return key; + } + + @Override + public String getDomain() { + return domain; + } + + // We cannot know the storage class currently + // @Override + // public String getStorageClass() { + // return storageClass; + // } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("MogileFileImpl [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append("]"); + return builder.toString(); + } + + @Override + public void copyToFile(File destination) throws IOException { + InputStream inputStream = null; + inputStream = getInputStream(); + // buffers internally and closes + FileUtils.copyInputStreamToFile(inputStream, destination); + } + + void setExecutor(Executor executor) { + this.executor = executor; + } + + private void unlockQuietly(Lock lock) { + try { + lock.unlock(); + } catch (IllegalMonitorStateException e) { + } + } + +} diff --git a/src/main/java/fm/last/moji/impl/MojiImpl.java b/src/main/java/fm/last/moji/impl/MojiImpl.java new file mode 100644 index 0000000..b52e3d4 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/MojiImpl.java @@ -0,0 +1,108 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.List; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFile; +import fm.last.moji.tracker.TrackerFactory; + +class MojiImpl implements Moji { + + private static final Logger log = LoggerFactory.getLogger(MojiImpl.class); + + private final TrackerFactory trackerFactory; + private final HttpConnectionFactory httpFactory; + private final String domain; + private final Executor executor; + + MojiImpl(TrackerFactory trackerFactory, HttpConnectionFactory httpFactory, String domain) { + this.domain = domain; + this.httpFactory = httpFactory; + this.trackerFactory = trackerFactory; + executor = new Executor(trackerFactory); + } + + @Override + public MojiFile getFile(String key) { + log.debug("new {}()", MojiFileImpl.class.getSimpleName()); + return new MojiFileImpl(key, domain, null, trackerFactory, httpFactory); + } + + @Override + public MojiFile getFile(String key, String storageClass) { + log.debug("new {}() with storage class", MojiFileImpl.class.getSimpleName()); + return new MojiFileImpl(key, domain, storageClass, trackerFactory, httpFactory); + } + + @Override + public void copyToMogile(File source, MojiFile destination) throws IOException { + OutputStream outputStream = null; + InputStream inputStream = new FileInputStream(source); + try { + outputStream = destination.getOutputStream(); + IOUtils.copy(inputStream, outputStream); // buffers internally + outputStream.flush(); + } finally { + IOUtils.closeQuietly(inputStream); + IOUtils.closeQuietly(outputStream); + } + } + + @Override + public List list(String keyPrefix) throws IOException { + log.debug("list() : {}", keyPrefix); + List list = null; + ListFilesCommand command = new ListFilesCommand(this, keyPrefix, domain); + executor.executeCommand(command); + list = command.getFileList(); + log.debug("list() -> {}", list); + return list; + } + + @Override + public List list(String keyPrefix, int limit) throws IOException { + log.debug("list() : {}, {}", keyPrefix, limit); + List list = null; + ListFilesCommand command = new ListFilesCommand(this, keyPrefix, domain, limit); + executor.executeCommand(command); + list = command.getFileList(); + log.debug("list() -> {}", list); + return list; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("MojiImpl [domain="); + builder.append(domain); + builder.append(", trackerFactory="); + builder.append(trackerFactory); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/PropertyMojiFactory.java b/src/main/java/fm/last/moji/impl/PropertyMojiFactory.java new file mode 100644 index 0000000..0875b6f --- /dev/null +++ b/src/main/java/fm/last/moji/impl/PropertyMojiFactory.java @@ -0,0 +1,131 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; +import java.io.InputStream; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.Properties; +import java.util.Set; + +import org.apache.commons.io.IOUtils; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFactory; +import fm.last.moji.tracker.TrackerFactory; +import fm.last.moji.tracker.impl.InetSocketAddressFactory; +import fm.last.moji.tracker.pool.MultiHostTrackerPool; + +/** + * Creates a {@link fm.last.moji.Moji Moji} instance using configuration information obtained from the following + * properties: + *

+ *

    + *
  • moji.domain
  • + *
  • moji.tracker.hosts
  • + *
+ *

+ * The properties are loaded from a /moji.properties classpath resource by default. The resource path can + * be specified by setting the moji.properties.resource.path system property. + */ +public class PropertyMojiFactory implements MojiFactory { + + public static final String RESOURCE_PATH_PROPERTY = "moji.properties.resource.path"; + private static final String DEFAULT_RESOURCE_PATH = "/moji.properties"; + + private static final String HOSTS_PROPERTY = "moji.tracker.hosts"; + private static final String DOMAIN_PROPERTY = "moji.domain"; + + private final Proxy proxy; + private volatile boolean initialised; + private String defaultDomain; + private TrackerFactory trackerFactory; + private HttpConnectionFactory httpFactory; + private final String propertiesPath; + + public PropertyMojiFactory(String propertiesPath, Proxy proxy) throws IOException { + this.propertiesPath = System.getProperty(RESOURCE_PATH_PROPERTY, propertiesPath); + this.proxy = proxy; + } + + public PropertyMojiFactory(String propertiesPath) throws IOException { + this(propertiesPath, Proxy.NO_PROXY); + } + + public PropertyMojiFactory(Proxy proxy) throws IOException { + this(DEFAULT_RESOURCE_PATH, proxy); + } + + public PropertyMojiFactory() throws IOException { + this(DEFAULT_RESOURCE_PATH, Proxy.NO_PROXY); + } + + @Override + public Moji getInstance() throws IOException { + initialise(); + return new MojiImpl(trackerFactory, httpFactory, defaultDomain); + } + + @Override + public Moji getInstance(String domain) throws IOException { + initialise(); + return new MojiImpl(trackerFactory, httpFactory, domain); + } + + private void initialise() throws IOException { + synchronized (this) { + if (!initialised) { + Properties properties = loadProperties(); + String addressesCsv = getHosts(properties); + defaultDomain = getDomain(properties); + Set addresses = InetSocketAddressFactory.newAddresses(addressesCsv); + + trackerFactory = new MultiHostTrackerPool(addresses, proxy); + httpFactory = new HttpConnectionFactory(trackerFactory.getProxy()); + initialised = true; + } + } + } + + private Properties loadProperties() throws IOException { + Properties properties = new Properties(); + InputStream stream = getClass().getResourceAsStream(propertiesPath); + try { + properties.load(stream); + } finally { + IOUtils.closeQuietly(stream); + } + return properties; + } + + private String getDomain(Properties properties) { + String domain = properties.getProperty(DOMAIN_PROPERTY); + if (domain == null || domain.isEmpty()) { + throw new IllegalStateException(DOMAIN_PROPERTY + " cannot be empty or null"); + } + return domain; + } + + private String getHosts(Properties properties) { + String host = properties.getProperty(HOSTS_PROPERTY); + if (host == null || host.isEmpty()) { + throw new IllegalStateException(HOSTS_PROPERTY + " cannot be empty or null"); + } + return host; + } + +} diff --git a/src/main/java/fm/last/moji/impl/RenameCommand.java b/src/main/java/fm/last/moji/impl/RenameCommand.java new file mode 100644 index 0000000..eec9ca6 --- /dev/null +++ b/src/main/java/fm/last/moji/impl/RenameCommand.java @@ -0,0 +1,52 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import fm.last.moji.tracker.Tracker; + +class RenameCommand implements MojiCommand { + + final String key; + final String domain; + final String newKey; + + RenameCommand(String key, String domain, String newKey) { + this.key = key; + this.domain = domain; + this.newKey = newKey; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + tracker.rename(key, domain, newKey); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("RenameCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", newKey="); + builder.append(newKey); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/impl/UpdateStorageClassCommand.java b/src/main/java/fm/last/moji/impl/UpdateStorageClassCommand.java new file mode 100644 index 0000000..884057e --- /dev/null +++ b/src/main/java/fm/last/moji/impl/UpdateStorageClassCommand.java @@ -0,0 +1,39 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import java.io.IOException; + +import fm.last.moji.tracker.Tracker; + +class UpdateStorageClassCommand implements MojiCommand { + + final String key; + final String domain; + final String newStorageClass; + + UpdateStorageClassCommand(String key, String domain, String newStorageClass) { + this.key = key; + this.domain = domain; + this.newStorageClass = newStorageClass; + } + + @Override + public void executeWithTracker(Tracker tracker) throws IOException { + tracker.updateStorageClass(key, domain, newStorageClass); + } + +} diff --git a/src/main/java/fm/last/moji/local/DefaultFileNamingStrategy.java b/src/main/java/fm/last/moji/local/DefaultFileNamingStrategy.java new file mode 100644 index 0000000..324aa08 --- /dev/null +++ b/src/main/java/fm/last/moji/local/DefaultFileNamingStrategy.java @@ -0,0 +1,93 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.local; + +import java.io.File; +import java.io.FilenameFilter; + +class DefaultFileNamingStrategy implements LocalFileNamingStrategy { + + private final File baseFolder; + + DefaultFileNamingStrategy(File baseFolder) { + this.baseFolder = baseFolder; + } + + @Override + public String newfileName(String domain, String key) { + return domain + "-" + key + ".dat"; + } + + @Override + public String domainForFileName(String fileName) { + int dashPosition = fileName.indexOf('-'); + String domain = fileName.substring(0, dashPosition); + return domain; + } + + @Override + public String keyForFileName(String fileName) { + int dashPosition = fileName.indexOf('-'); + int periodPosition = fileName.lastIndexOf('.'); + String key = fileName.substring(dashPosition + 1, periodPosition); + return key; + } + + @Override + public File folderForDomain(String domain) { + return baseFolder; + } + + @Override + public FilenameFilter filterForPrefix(String domain, final String keyPrefix) { + return new KeyPrefixFileNameFilter(domain, keyPrefix); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("DefaultFileNamingStrategy [baseFolder="); + builder.append(baseFolder); + builder.append("]"); + return builder.toString(); + } + + private final class KeyPrefixFileNameFilter implements FilenameFilter { + private final String keyPrefix; + private final String domain; + + private KeyPrefixFileNameFilter(String domain, String keyPrefix) { + this.domain = domain; + this.keyPrefix = keyPrefix; + } + + @Override + public boolean accept(File dir, String name) { + if (!baseFolder.equals(dir)) { + return false; + } + if (name.startsWith(".")) { + return false; + } + if (!name.startsWith(domain + "-")) { + return false; + } + String key = keyForFileName(name); + return key.startsWith(keyPrefix); + } + } + +} diff --git a/src/main/java/fm/last/moji/local/LocalFileNamingStrategy.java b/src/main/java/fm/last/moji/local/LocalFileNamingStrategy.java new file mode 100644 index 0000000..6253973 --- /dev/null +++ b/src/main/java/fm/last/moji/local/LocalFileNamingStrategy.java @@ -0,0 +1,37 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.local; + +import java.io.File; +import java.io.FilenameFilter; + +/** + * Used by {@link fm.last.moji.local.LocalFileSystemMoji LocalFileSystemMoji} to generate and resolve local filenames + * for given keys and domains. + */ +public interface LocalFileNamingStrategy { + + String newfileName(String domain, String key); + + String domainForFileName(String fileName); + + String keyForFileName(String fileName); + + File folderForDomain(String domain); + + FilenameFilter filterForPrefix(String domain, String keyPrefix); + +} diff --git a/src/main/java/fm/last/moji/local/LocalFileSystemMoji.java b/src/main/java/fm/last/moji/local/LocalFileSystemMoji.java new file mode 100644 index 0000000..dfbe019 --- /dev/null +++ b/src/main/java/fm/last/moji/local/LocalFileSystemMoji.java @@ -0,0 +1,122 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.local; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.io.FileUtils; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFile; + +/** + * A simple {@link fm.last.moji.Moji Moji} implementation that uses the local filesystem for storage. This is intended + * for testing only. + */ +public class LocalFileSystemMoji implements Moji { + + private final File baseFolder; + private final LocalFileNamingStrategy namingStrategy; + private final String domain; + + public LocalFileSystemMoji(File baseFolder, String domain) { + this(baseFolder, domain, new DefaultFileNamingStrategy(baseFolder)); + } + + public LocalFileSystemMoji(File baseFolder, String domain, LocalFileNamingStrategy namingStrategy) { + createBaseFolderIfNeeded(baseFolder); + this.baseFolder = baseFolder; + this.domain = domain; + this.namingStrategy = namingStrategy; + } + + public File getBaseFolder() { + return baseFolder; + } + + public LocalFileNamingStrategy getNamingStrategy() { + return namingStrategy; + } + + public String getDomain() { + return domain; + } + + @Override + public MojiFile getFile(String key) { + return new LocalMojiFile(namingStrategy, baseFolder, domain, key); + } + + @Override + public MojiFile getFile(String key, String storageClass) { + return new LocalMojiFile(namingStrategy, baseFolder, domain, key); + } + + @Override + public void copyToMogile(File source, MojiFile destination) throws IOException { + LocalMojiFile localDestination = (LocalMojiFile) destination; + FileUtils.copyFile(source, localDestination.file); + } + + @Override + public List list(final String keyPrefix) { + File[] files = baseFolder.listFiles(namingStrategy.filterForPrefix(domain, keyPrefix)); + List mojiFiles = new ArrayList(files.length); + for (File file : files) { + String key = namingStrategy.keyForFileName(file.getName()); + mojiFiles.add(new LocalMojiFile(namingStrategy, baseFolder, domain, key)); + } + return null; + } + + @Override + public List list(String keyPrefix, int limit) { + List list = list(keyPrefix); + int count = limit > list.size() ? list.size() : limit; + List mojiFiles = new ArrayList(); + for (int i = 0; i < count; i++) { + mojiFiles.add(list.get(i)); + } + return mojiFiles; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("LocalFileSystemMoji [baseFolder="); + builder.append(baseFolder); + builder.append(", domain="); + builder.append(domain); + builder.append(", namingStrategy="); + builder.append(namingStrategy); + builder.append("]"); + return builder.toString(); + } + + private void createBaseFolderIfNeeded(File baseFolder) { + boolean exists = baseFolder.exists(); + if (!exists) { + boolean mkdirs = baseFolder.mkdirs(); + if (!mkdirs) { + throw new IllegalStateException("Could not create base directory: " + baseFolder.getAbsolutePath()); + } + } + } + +} diff --git a/src/main/java/fm/last/moji/local/LocalMojiFile.java b/src/main/java/fm/last/moji/local/LocalMojiFile.java new file mode 100644 index 0000000..838e18c --- /dev/null +++ b/src/main/java/fm/last/moji/local/LocalMojiFile.java @@ -0,0 +1,136 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.local; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.URL; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.io.FileUtils; + +import fm.last.moji.MojiFile; + +class LocalMojiFile implements MojiFile { + + private final String domain; + private final File baseDir; + final File file; + private String key; + private final LocalFileNamingStrategy namingStrategy; + + LocalMojiFile(LocalFileNamingStrategy namingStrategy, File baseDir, String domain, String key) { + this.namingStrategy = namingStrategy; + this.baseDir = baseDir; + this.key = key; + this.domain = domain; + file = new File(baseDir, namingStrategy.newfileName(domain, key)); + } + + @Override + public boolean exists() throws IOException { + return file.exists(); + } + + @Override + public void delete() throws IOException { + if (!file.exists()) { + throw new FileNotFoundException(file.getCanonicalPath()); + } + file.delete(); + } + + @Override + public InputStream getInputStream() throws IOException { + if (!file.exists()) { + throw new FileNotFoundException(file.getCanonicalPath()); + } + return new FileInputStream(file); + } + + @Override + public OutputStream getOutputStream() throws IOException { + if (!file.exists()) { + file.createNewFile(); + } + return new FileOutputStream(file); + } + + @Override + public void copyToFile(File destination) throws IOException { + if (!file.exists()) { + throw new FileNotFoundException(file.getCanonicalPath()); + } + FileUtils.copyFile(file, destination); + } + + @Override + public long length() throws IOException { + if (!file.exists()) { + throw new FileNotFoundException(file.getCanonicalPath()); + } + return file.length(); + } + + @Override + public void rename(String newKey) throws IOException { + if (!file.exists()) { + throw new FileNotFoundException(file.getCanonicalPath()); + } + file.renameTo(new File(baseDir, namingStrategy.newfileName(domain, newKey))); + key = newKey; + } + + @Override + public List getPaths() throws IOException { + return Collections.singletonList(file.toURI().toURL()); + } + + @Override + public String getKey() { + return key; + } + + @Override + public String getDomain() { + return domain; + } + + @Override + public void modifyStorageClass(String storageClass) throws IOException { + // ignored + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("LocalMogileFile [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", file="); + builder.append(file); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/spring/SpringMojiBean.java b/src/main/java/fm/last/moji/spring/SpringMojiBean.java new file mode 100644 index 0000000..4c7f638 --- /dev/null +++ b/src/main/java/fm/last/moji/spring/SpringMojiBean.java @@ -0,0 +1,180 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.spring; + +import java.io.File; +import java.io.IOException; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.List; +import java.util.Set; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFile; +import fm.last.moji.impl.DefaultMojiFactory; +import fm.last.moji.tracker.impl.InetSocketAddressFactory; +import fm.last.moji.tracker.pool.MultiHostTrackerPool; + +/** + * A {@link fm.last.moji.Moji Moji} delegate that exposes pool properties and is easily configured in Spring. + */ +public class SpringMojiBean implements Moji { + + private final Moji moji; + private final MultiHostTrackerPool poolingTrackerFactory; + + public SpringMojiBean(String addressesCsv, String domain) { + this(addressesCsv, Proxy.NO_PROXY, domain); + } + + public SpringMojiBean(String addressesCsv, Proxy proxy, String domain) { + Set addresses = InetSocketAddressFactory.newAddresses(addressesCsv); + poolingTrackerFactory = new MultiHostTrackerPool(addresses, proxy); + DefaultMojiFactory factory = new DefaultMojiFactory(poolingTrackerFactory, domain); + moji = factory.getInstance(); + } + + @Override + public MojiFile getFile(String key) { + return moji.getFile(key); + } + + @Override + public MojiFile getFile(String key, String storageClass) { + return moji.getFile(key, storageClass); + } + + @Override + public void copyToMogile(File source, MojiFile destination) throws IOException { + moji.copyToMogile(source, destination); + } + + @Override + public List list(String keyPrefix) throws IOException { + return moji.list(keyPrefix); + } + + @Override + public List list(String keyPrefix, int limit) throws IOException { + return moji.list(keyPrefix, limit); + } + + /** + * See: {@link fm.last.moji.tracker.TrackerFactory#getProxy()} + */ + public Proxy getProxy() { + return poolingTrackerFactory.getProxy(); + } + + /** + * See: {@link fm.last.moji.tracker.TrackerFactory#getAddresses()} + */ + public Set getAddresses() { + return poolingTrackerFactory.getAddresses(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#close()} + */ + public void close() throws Exception { + poolingTrackerFactory.close(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxActive()} + */ + public int getMaxActive() { + return poolingTrackerFactory.getMaxActive(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setMaxActive(int)} + */ + public void setMaxActive(int maxActive) { + poolingTrackerFactory.setMaxActive(maxActive); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxWait()} + */ + public long getMaxWait() { + return poolingTrackerFactory.getMaxWait(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setMaxWait(long)} + */ + public void setMaxWait(long maxWait) { + poolingTrackerFactory.setMaxWait(maxWait); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxIdle()} + */ + public int getMaxIdle() { + return poolingTrackerFactory.getMaxIdle(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setMaxIdle(int)} + */ + public void setMaxIdle(int maxIdle) { + poolingTrackerFactory.setMaxIdle(maxIdle); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getTestOnBorrow()} + */ + public boolean getTestOnBorrow() { + return poolingTrackerFactory.getTestOnBorrow(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setTestOnBorrow(boolean)} + */ + public void setTestOnBorrow(boolean testOnBorrow) { + poolingTrackerFactory.setTestOnBorrow(testOnBorrow); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getTestOnReturn()} + */ + public boolean getTestOnReturn() { + return poolingTrackerFactory.getTestOnReturn(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setTestOnReturn(boolean)} + */ + public void setTestOnReturn(boolean testOnReturn) { + poolingTrackerFactory.setTestOnReturn(testOnReturn); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getNumActive()} + */ + public int getNumActive() { + return poolingTrackerFactory.getNumActive(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getNumIdle()} + */ + public int getNumIdle() { + return poolingTrackerFactory.getNumIdle(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/Destination.java b/src/main/java/fm/last/moji/tracker/Destination.java new file mode 100644 index 0000000..b9193d8 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/Destination.java @@ -0,0 +1,60 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +import java.net.URL; + +/** + * Represents a MogileFS remote file location. + */ +public class Destination { + + private final URL path; + private final int devId; + private final int fid; + + public Destination(URL path, int devId, int fid) { + this.path = path; + this.devId = devId; + this.fid = fid; + } + + public URL getPath() { + return path; + } + + public int getDevId() { + return devId; + } + + public int getFid() { + return fid; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("Destination [path="); + builder.append(path); + builder.append(", devId="); + builder.append(devId); + builder.append(", fid="); + builder.append(fid); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/KeyExistsAlreadyException.java b/src/main/java/fm/last/moji/tracker/KeyExistsAlreadyException.java new file mode 100644 index 0000000..ade78ae --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/KeyExistsAlreadyException.java @@ -0,0 +1,41 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +/** + * An attempt was made to assign an existing key to a new {@link fm.last.moji.MojiFile}. + */ +public class KeyExistsAlreadyException extends TrackerException { + + private static final long serialVersionUID = 1L; + private final String domain; + private final String key; + + public KeyExistsAlreadyException(String domain, String key) { + super("domain=" + domain + ",key=" + key); + this.domain = domain; + this.key = key; + } + + public String getDomain() { + return domain; + } + + public String getKey() { + return key; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/Tracker.java b/src/main/java/fm/last/moji/tracker/Tracker.java new file mode 100644 index 0000000..f96edc8 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/Tracker.java @@ -0,0 +1,82 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +import java.net.URL; +import java.util.List; + +/** + * Service interface of the MogileFS 'backend'. + * + * @see The Perl MogileFS + * client. + */ +public interface Tracker { + + List getPaths(String key, String domain) throws TrackerException; + + List createOpen(String key, String domain, String storageClass) throws TrackerException; + + void createClose(String key, String domain, Destination destination, long size) throws TrackerException; + + /** + * Delete a key from MogileFS in the given domain. + * + * @param key The key to delete. + * @param domain The domain in which the key resides. + * @throws TrackerException If there was a problem deleting the key. + */ + void delete(String key, String domain) throws TrackerException; + + /** + * Rename file (key) in MogileFS from oldKey to newKey. + * + * @param oldKey The key to rename. + * @param domain The domain in which the old key resides. + * @param domain The new key. + * @throws TrackerException If there was a problem deleting the key. + */ + void rename(String oldKey, String domain, String newKey) throws TrackerException; + + /** + * Update the storage class of a pre-existing file, causing the file to become more or less replicated + * + * @param key The key of the file to modify. + * @param domain The domain in which the key resides. + * @param newStorageClass The new storage class. + * @throws TrackerException If there was a problem updaing the storage class. + */ + void updateStorageClass(String key, String domain, String newStorageClass) throws TrackerException; + + void noop() throws TrackerException; + + /** + * Closes the resources used by this tracker. Pooled implementations may just return the tracker to the pool. + */ + void close(); + + /** + * Get a list of keys matching a given prefix in the target domain. + * + * @param domain The domain in which to perform the key search. + * @param keyPrefix The key prefix to match against. + * @param limit The maximim number of matches to return. + * @return A list of matched keys, or an empty list if there were no matches. + * @throws TrackerException If there was a problem matching. + */ + List list(String domain, String keyPrefix, Integer limit) throws TrackerException; + +} \ No newline at end of file diff --git a/src/main/java/fm/last/moji/tracker/TrackerException.java b/src/main/java/fm/last/moji/tracker/TrackerException.java new file mode 100644 index 0000000..e8befad --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/TrackerException.java @@ -0,0 +1,43 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +import java.io.IOException; + +/** + * General problem when using a {@link fm.last.moji.tracker.Tracker Tracker}. + */ +public class TrackerException extends IOException { + + private static final long serialVersionUID = 1L; + + public TrackerException() { + super(); + } + + public TrackerException(String message, Throwable cause) { + super(message, cause); + } + + public TrackerException(String message) { + super(message); + } + + public TrackerException(Throwable cause) { + super(cause); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/TrackerFactory.java b/src/main/java/fm/last/moji/tracker/TrackerFactory.java new file mode 100644 index 0000000..286cd42 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/TrackerFactory.java @@ -0,0 +1,49 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.Set; + +/** + * Provides usable {@link Tracker} instances for communicating with the MogileFS 'backend'. + */ +public interface TrackerFactory { + + /** + * Gets a new usable {@link Tracker}. + * + * @return A valid tracker. + * @throws TrackerException If there was a problem obtaining a tracker. + */ + Tracker getTracker() throws TrackerException; + + /** + * The host addresses of {@link Tracker Trackers} that this factory may return. + * + * @return A set of host addresses. + */ + Set getAddresses(); + + /** + * The network proxy used by the {@link Tracker Trackers} returned from this factory. + * + * @return The proxy, or {@link Proxy#NO_PROXY NO_PROXY} if no proxy has been set. + */ + Proxy getProxy(); + +} diff --git a/src/main/java/fm/last/moji/tracker/UnknownKeyException.java b/src/main/java/fm/last/moji/tracker/UnknownKeyException.java new file mode 100644 index 0000000..4e33897 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/UnknownKeyException.java @@ -0,0 +1,41 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +/** + * An attempt was made to read from or modify a non-existent {@link fm.last.moji.MojiFile}. + */ +public class UnknownKeyException extends TrackerException { + + private static final long serialVersionUID = 1L; + private final String domain; + private final String key; + + public UnknownKeyException(String domain, String key) { + super("domain=" + domain + ",key=" + key); + this.domain = domain; + this.key = key; + } + + public String getDomain() { + return domain; + } + + public String getKey() { + return key; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/UnknownStorageClassException.java b/src/main/java/fm/last/moji/tracker/UnknownStorageClassException.java new file mode 100644 index 0000000..dcd7bc4 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/UnknownStorageClassException.java @@ -0,0 +1,36 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker; + +/** + * An attempt was made to assign an non-existent storage class to a {@link fm.last.moji.MojiFile}. + */ +public class UnknownStorageClassException extends TrackerException { + + private static final long serialVersionUID = 1L; + + private final String storageClass; + + public UnknownStorageClassException(String storageClass) { + super("storageClass=" + storageClass); + this.storageClass = storageClass; + } + + public String getStorageClass() { + return storageClass; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/AbstractTrackerFactory.java b/src/main/java/fm/last/moji/tracker/impl/AbstractTrackerFactory.java new file mode 100644 index 0000000..6da1c38 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/AbstractTrackerFactory.java @@ -0,0 +1,84 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.net.Socket; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; + +/** + * Provides some common {@link fm.last.moji.tracker.TrackerFactory TrackerFactory} functionality. + */ +public class AbstractTrackerFactory { + + private static final Logger log = LoggerFactory.getLogger(AbstractTrackerFactory.class); + + private final Proxy proxy; + + public AbstractTrackerFactory(Proxy proxy) { + this.proxy = proxy; + } + + public Tracker newTracker(InetSocketAddress newAddress) throws TrackerException { + log.debug("new {}()", TrackerImpl.class.getSimpleName()); + Tracker tracker = null; + BufferedReader reader = null; + Writer writer = null; + Socket socket = null; + try { + socket = new Socket(proxy); + log.debug("Connecting to: {}:", newAddress, socket.getPort()); + socket.connect(newAddress); + reader = new BufferedReader(new InputStreamReader(socket.getInputStream())); + writer = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream())); + RequestHandler requestHandler = new RequestHandler(writer, reader); + tracker = new TrackerImpl(socket, requestHandler); + } catch (IOException e) { + IOUtils.closeQuietly(reader); + IOUtils.closeQuietly(writer); + IOUtils.closeQuietly(socket); + throw new TrackerException(e); + } + return tracker; + } + + public Proxy getProxy() { + return proxy; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("AbstractTrackerFactory [proxy="); + builder.append(proxy); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/Charsets.java b/src/main/java/fm/last/moji/tracker/impl/Charsets.java new file mode 100644 index 0000000..5c92820 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/Charsets.java @@ -0,0 +1,25 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +enum Charsets { + UTF_8; + + public String value() { + return "UTF-8"; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/CommunicationException.java b/src/main/java/fm/last/moji/tracker/impl/CommunicationException.java new file mode 100644 index 0000000..015f9bc --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/CommunicationException.java @@ -0,0 +1,43 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import fm.last.moji.tracker.TrackerException; + +/** + * Communication problem when using a {@link fm.last.moji.tracker.Tracker Tracker}. + */ +public class CommunicationException extends TrackerException { + + private static final long serialVersionUID = 1L; + + public CommunicationException() { + super(); + } + + CommunicationException(String message, Throwable cause) { + super(message, cause); + } + + CommunicationException(String message) { + super(message); + } + + CommunicationException(Throwable cause) { + super(cause); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/CreateOpenOperation.java b/src/main/java/fm/last/moji/tracker/impl/CreateOpenOperation.java new file mode 100644 index 0000000..b02a330 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/CreateOpenOperation.java @@ -0,0 +1,113 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.ErrorCode.UNKNOWN_CLASS; +import static fm.last.moji.tracker.impl.ErrorCode.UNKNOWN_KEY; +import static fm.last.moji.tracker.impl.ResponseStatus.OK; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.UnknownStorageClassException; + +class CreateOpenOperation { + private final String domain; + private final String key; + private final String storageClass; + private final boolean multipleDestinations; + private final RequestHandler requestHandler; + + private List destinations; + + CreateOpenOperation(RequestHandler requestHandler, String domain, String key, String storageClass, + boolean multipleDestinations) { + this.requestHandler = requestHandler; + this.domain = domain; + this.key = key; + this.storageClass = storageClass; + this.multipleDestinations = multipleDestinations; + destinations = Collections.emptyList(); + } + + public void execute() throws TrackerException { + Request request = buildRequest(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + if (UNKNOWN_CLASS.isContainedInLine(response.getMessage())) { + throw new UnknownStorageClassException(storageClass); + } + if (!UNKNOWN_KEY.isContainedInLine(response.getMessage())) { + throw new TrackerException(response.getMessage()); + } + } else { + extractReturnValues(response); + } + } + + List getDestinations() { + return destinations; + } + + private Request buildRequest() { + Request.Builder builder = new Request.Builder(4).command("create_open").arg("domain", domain).arg("key", key) + .arg("multi_dest", multipleDestinations); + if (storageClass != null && !storageClass.isEmpty()) { + builder.arg("class", storageClass); + } + Request request = builder.build(); + return request; + } + + private void extractReturnValues(Response response) throws TrackerException { + int fid = Integer.parseInt(response.getValue("fid")); + int pathCount = Integer.parseInt(response.getValue("dev_count")); + destinations = new ArrayList(pathCount); + + for (int i = 1; i <= pathCount; i++) { + URL url; + Integer devId; + try { + url = new URL(response.getValue("path_" + i)); + devId = Integer.valueOf(response.getValue("devid_" + i)); + destinations.add(new Destination(url, devId, fid)); + } catch (MalformedURLException e) { + throw new TrackerException(e); + } + } + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("CreateOpenCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", storageClass="); + builder.append(storageClass); + builder.append(", multipleDestinations="); + builder.append(multipleDestinations); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/ErrorCode.java b/src/main/java/fm/last/moji/tracker/impl/ErrorCode.java new file mode 100644 index 0000000..a2bae48 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/ErrorCode.java @@ -0,0 +1,44 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.util.Arrays; +import java.util.HashSet; +import java.util.Set; + +enum ErrorCode { + UNKNOWN_KEY("unknown_key"), KEY_EXISTS("key_exists"), UNKNOWN_CLASS("unreg_class", "class_not_found"), NONE_MATCH( + "none_match"); + + private Set messages; + + private ErrorCode(String... messages) { + this.messages = new HashSet(Arrays.asList(messages)); + } + + boolean isContainedInLine(String line) { + if (line == null || line.isEmpty()) { + return false; + } + for (String message : messages) { + if (line.contains(message)) { + return true; + } + } + return false; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/GetPathsOperation.java b/src/main/java/fm/last/moji/tracker/impl/GetPathsOperation.java new file mode 100644 index 0000000..38ee4eb --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/GetPathsOperation.java @@ -0,0 +1,92 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.ErrorCode.UNKNOWN_KEY; +import static fm.last.moji.tracker.impl.ResponseStatus.OK; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.UnknownKeyException; + +class GetPathsOperation { + + private final String domain; + private final String key; + private final boolean verify; + private final RequestHandler requestHandler; + private List paths; + + GetPathsOperation(RequestHandler requestHandler, String domain, String key, boolean verify) { + this.domain = domain; + this.key = key; + this.verify = verify; + this.requestHandler = requestHandler; + paths = Collections.emptyList(); + } + + public void execute() throws TrackerException { + Request request = new Request.Builder(3).command("get_paths").arg("domain", domain).arg("key", key) + .arg("noverify", !verify).build(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + if (UNKNOWN_KEY.isContainedInLine(response.getMessage())) { + throw new UnknownKeyException(domain, key); + } + throw new TrackerException(response.getMessage()); + } else { + paths = extractReturnValue(response); + } + } + + List getPaths() { + return paths; + } + + private List extractReturnValue(Response response) throws TrackerException { + int pathCount = Integer.parseInt(response.getValue("paths")); + List urls = new ArrayList(pathCount); + for (int i = 1; i <= pathCount; i++) { + URL url; + try { + url = new URL(response.getValue("path" + i)); + urls.add(url); + } catch (MalformedURLException e) { + throw new TrackerException(e); + } + } + return urls; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("GetPathsCommand [domain="); + builder.append(domain); + builder.append(", key="); + builder.append(key); + builder.append(", verify="); + builder.append(verify); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/InetSocketAddressFactory.java b/src/main/java/fm/last/moji/tracker/impl/InetSocketAddressFactory.java new file mode 100644 index 0000000..3e881bd --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/InetSocketAddressFactory.java @@ -0,0 +1,56 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.net.UnknownHostException; +import java.util.HashSet; +import java.util.Set; + +public class InetSocketAddressFactory { + + private InetSocketAddressFactory() { + } + + public static Set newAddresses(String addressesCsv) { + Set addresses = new HashSet(); + for (String addressElement : addressesCsv.split(",")) { + addresses.add(addressElement.trim()); + } + Set socketAddresses = new HashSet(); + for (String address : addresses) { + InetSocketAddress socketAddress = newAddress(address); + socketAddresses.add(socketAddress); + } + return socketAddresses; + } + + public static InetSocketAddress newAddress(String addressString) { + String[] parts = addressString.split(":"); + String host = parts[0]; + int port = Integer.valueOf(parts[1]); + InetAddress address; + try { + address = InetAddress.getByName(host); + } catch (UnknownHostException e) { + throw new IllegalArgumentException("Invalid ':': '" + addressString + "'", e); + } + InetSocketAddress socketAddress = new InetSocketAddress(address, port); + return socketAddress; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/ListKeysOperation.java b/src/main/java/fm/last/moji/tracker/impl/ListKeysOperation.java new file mode 100644 index 0000000..3313e84 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/ListKeysOperation.java @@ -0,0 +1,91 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.ErrorCode.NONE_MATCH; +import static fm.last.moji.tracker.impl.ResponseStatus.OK; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import fm.last.moji.tracker.TrackerException; + +class ListKeysOperation { + + private final RequestHandler requestHandler; + private final String domain; + private final String keyPrefix; + private final Integer limit; + private List keys; + + ListKeysOperation(RequestHandler requestHandler, String domain, String keyPrefix, Integer limit) { + this.requestHandler = requestHandler; + this.domain = domain; + this.keyPrefix = keyPrefix; + this.limit = limit; + keys = Collections.emptyList(); + } + + void execute() throws TrackerException { + Request request = buildRequest(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + if (!NONE_MATCH.isContainedInLine(response.getMessage())) { + throw new TrackerException(response.getMessage()); + } + } else { + keys = extractReturnValue(response); + } + } + + List getKeys() { + return keys; + } + + private Request buildRequest() { + Request.Builder builder = new Request.Builder(3).command("list_keys").arg("domain", domain) + .arg("prefix", keyPrefix); + if (limit != null) { + builder.arg("limit", limit); + } + Request request = builder.build(); + return request; + } + + private List extractReturnValue(Response response) { + int keyCount = Integer.parseInt(response.getValue("key_count")); + List keys = new ArrayList(); + for (int i = 1; i <= keyCount; i++) { + keys.add(response.getValue("key_" + i)); + } + return keys; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("ListKeysOperation [domain="); + builder.append(domain); + builder.append(", keyPrefix="); + builder.append(keyPrefix); + builder.append(", limit="); + builder.append(limit); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/Request.java b/src/main/java/fm/last/moji/tracker/impl/Request.java new file mode 100644 index 0000000..169e35c --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/Request.java @@ -0,0 +1,132 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.Charsets.UTF_8; + +import java.io.IOException; +import java.io.UnsupportedEncodingException; +import java.io.Writer; +import java.net.URL; +import java.net.URLEncoder; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +class Request { + + private static final Logger log = LoggerFactory.getLogger(Request.class); + + private final String command; + private final Map arguments; + + private Request(Builder builder) { + command = builder.command; + arguments = builder.arguments; + } + + String getCommand() { + return command; + } + + Map getArguments() { + return arguments; + } + + void writeTo(Writer writer) throws IOException { + StringBuilder wire = new StringBuilder(); + wire.append(command); + wire.append(' '); + boolean first = true; + for (Entry entry : arguments.entrySet()) { + if (first) { + first = false; + } else { + wire.append('&'); + } + try { + wire.append(URLEncoder.encode(entry.getKey(), UTF_8.value())); + wire.append('='); + wire.append(URLEncoder.encode(entry.getValue(), UTF_8.value())); + } catch (UnsupportedEncodingException ignored) { + } + } + wire.append('\r'); + wire.append('\n'); + log.debug("Sent: {}", wire); + writer.write(wire.toString()); + writer.flush(); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("Request [command="); + builder.append(command); + builder.append(", arguments="); + builder.append(arguments); + builder.append("]"); + return builder.toString(); + } + + static class Builder { + + private final Map arguments; + private String command; + + Builder(int expectedSize) { + arguments = new HashMap(expectedSize); + } + + Builder command(String command) { + this.command = command; + return this; + } + + Builder arg(String key, String value) { + arguments.put(key, value); + return this; + } + + Builder arg(String key, int value) { + arguments.put(key, Integer.toString(value)); + return this; + } + + Builder arg(String key, long value) { + arguments.put(key, Long.toString(value)); + return this; + } + + Builder arg(String key, boolean value) { + arguments.put(key, value ? "1" : "0"); + return this; + } + + Builder arg(String key, URL value) { + arguments.put(key, value.toString()); + return this; + } + + Request build() { + return new Request(this); + } + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/RequestHandler.java b/src/main/java/fm/last/moji/tracker/impl/RequestHandler.java new file mode 100644 index 0000000..37c8a7a --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/RequestHandler.java @@ -0,0 +1,79 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.Writer; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.TrackerException; + +class RequestHandler { + + private static final Logger log = LoggerFactory.getLogger(RequestHandler.class); + + private final Writer writer; + private final BufferedReader reader; + + RequestHandler(Writer writer, BufferedReader reader) { + this.writer = writer; + this.reader = reader; + } + + Response performRequest(Request request) throws CommunicationException { + Response response = null; + String line = null; + + try { + log.debug("{}", request); + request.writeTo(writer); + line = reader.readLine(); + log.debug("Read: {}", line); + response = createResponseFromLine(line); + log.debug("{}", response); + } catch (IOException e) { + throw new CommunicationException(e); + } + return response; + } + + void close() { + IOUtils.closeQuietly(reader); + IOUtils.closeQuietly(writer); + } + + private Response createResponseFromLine(String line) throws TrackerException { + if (line == null) { + throw new TrackerException("Empty response from tracker"); + } + int firstSpace = line.indexOf(' '); + if (firstSpace < 0) { + throw new TrackerException("Invalid response from tracker: '" + line + "'"); + } + ResponseStatus status = ResponseStatus.valueOfCode(line.substring(0, firstSpace)); + if (status == null) { + throw new TrackerException("Invalid response from tracker: '" + line + "'"); + } + String payload = line.substring(firstSpace + 1); + Response response = new Response(status, payload); + return response; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/Response.java b/src/main/java/fm/last/moji/tracker/impl/Response.java new file mode 100644 index 0000000..482003d --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/Response.java @@ -0,0 +1,95 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.Charsets.UTF_8; + +import java.io.UnsupportedEncodingException; +import java.net.URLDecoder; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +class Response { + + private static final Logger log = LoggerFactory.getLogger(Response.class); + + private final Map values; + private final ResponseStatus status; + private final String message; + + Response(ResponseStatus status, String payload) { + this.status = status; + if (status == ResponseStatus.OK) { + values = decodePayload(payload); + message = null; + } else { + message = payload; + values = Collections.emptyMap(); + } + } + + ResponseStatus getStatus() { + return status; + } + + String getValue(String key) { + return values.get(key); + } + + String getMessage() { + return message; + } + + private Map decodePayload(String encoded) { + HashMap map = new HashMap(); + try { + if (encoded == null || encoded.length() == 0) { + return map; + } + String[] parts = encoded.split("&"); + for (String part : parts) { + String[] pair = part.split("="); + if (pair.length != 2) { + log.error("Poorly encoded string: {} ", encoded); + continue; + } + map.put(pair[0], URLDecoder.decode(pair[1], UTF_8.value())); + } + return map; + } catch (UnsupportedEncodingException e) { + log.error("Problem decoding response", e); + return null; + } + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("Response [status="); + builder.append(status); + builder.append(", values="); + builder.append(values); + builder.append(", message="); + builder.append(message); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/ResponseStatus.java b/src/main/java/fm/last/moji/tracker/impl/ResponseStatus.java new file mode 100644 index 0000000..08b16bc --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/ResponseStatus.java @@ -0,0 +1,46 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.util.HashMap; +import java.util.Map; + +enum ResponseStatus { + OK("OK"), ERROR("ERR"); + + private static Map wireCodeToVal = new HashMap(); + + static { + for (ResponseStatus status : ResponseStatus.values()) { + wireCodeToVal.put(status.value(), status); + } + } + + private final String code; + + private ResponseStatus(String code) { + this.code = code; + } + + String value() { + return code; + } + + static ResponseStatus valueOfCode(String code) { + return wireCodeToVal.get(code); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/SingleHostTrackerFactory.java b/src/main/java/fm/last/moji/tracker/impl/SingleHostTrackerFactory.java new file mode 100644 index 0000000..c4682ca --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/SingleHostTrackerFactory.java @@ -0,0 +1,73 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.Collections; +import java.util.Set; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.TrackerFactory; + +/** + * Single host tracker factory that creates a new connection on each {@link #getTracker()} request. Do not use this + * directly - only as a building block for pooling {@link TrackerFactory TrackerFactorys}. + */ +public class SingleHostTrackerFactory implements TrackerFactory { + + private final AbstractTrackerFactory delegateFactory; + private final InetSocketAddress address; + + /** + * Creates a tracker factory for the given host address and use the supplied network proxy. + * + * @param addresses Tracker host address + * @param proxy Network proxy - use Proxy.NO_PROXY if a proxy isn't needed. + */ + public SingleHostTrackerFactory(InetSocketAddress address, Proxy proxy) { + delegateFactory = new AbstractTrackerFactory(proxy); + this.address = address; + } + + @Override + public Tracker getTracker() throws TrackerException { + return delegateFactory.newTracker(address); + } + + @Override + public Set getAddresses() { + return Collections.singleton(address); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("SingletonTrackerFactory [address="); + builder.append(address); + builder.append(", proxy="); + builder.append(getProxy()); + builder.append("]"); + return builder.toString(); + } + + @Override + public Proxy getProxy() { + return delegateFactory.getProxy(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/impl/TrackerImpl.java b/src/main/java/fm/last/moji/tracker/impl/TrackerImpl.java new file mode 100644 index 0000000..eddc7f1 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/impl/TrackerImpl.java @@ -0,0 +1,169 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static fm.last.moji.tracker.impl.ErrorCode.KEY_EXISTS; +import static fm.last.moji.tracker.impl.ErrorCode.UNKNOWN_CLASS; +import static fm.last.moji.tracker.impl.ErrorCode.UNKNOWN_KEY; +import static fm.last.moji.tracker.impl.ResponseStatus.OK; + +import java.net.Socket; +import java.net.URL; +import java.util.List; + +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.KeyExistsAlreadyException; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.UnknownKeyException; +import fm.last.moji.tracker.UnknownStorageClassException; +import fm.last.moji.tracker.impl.Request.Builder; + +class TrackerImpl implements Tracker { + + private static final Logger log = LoggerFactory.getLogger(TrackerImpl.class); + + private final Socket socket; + private final RequestHandler requestHandler; + + public TrackerImpl(Socket socket, RequestHandler requestHandler) { + this.socket = socket; + this.requestHandler = requestHandler; + } + + @Override + public List getPaths(String key, String domain) throws TrackerException { + GetPathsOperation operation = new GetPathsOperation(requestHandler, domain, key, false); + operation.execute(); + return operation.getPaths(); + } + + @Override + public List createOpen(String key, String domain, String storageClass) throws TrackerException { + CreateOpenOperation operation = new CreateOpenOperation(requestHandler, domain, key, storageClass, true); + operation.execute(); + return operation.getDestinations(); + } + + @Override + public List list(String domain, String keyPrefix, Integer limit) throws TrackerException { + ListKeysOperation operation = new ListKeysOperation(requestHandler, domain, keyPrefix, limit); + operation.execute(); + return operation.getKeys(); + } + + @Override + public void createClose(String key, String domain, Destination destination, long size) throws TrackerException { + Request request = new Builder(6).command("create_close").arg("domain", domain).arg("key", key) + .arg("devid", destination.getDevId()).arg("path", destination.getPath()).arg("fid", destination.getFid()) + .arg("size", size).build(); + Response response = requestHandler.performRequest(request); + handleGeneralResponseError(response); + } + + @Override + public void delete(String key, String domain) throws TrackerException { + Request request = new Request.Builder(2).command("delete").arg("domain", domain).arg("key", key).build(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + String message = response.getMessage(); + handleUnknownKeyException(key, domain, message); + throw new TrackerException(message); + } + } + + @Override + public void rename(String fromKey, String domain, String toKey) throws TrackerException { + Request request = new Request.Builder(3).command("rename").arg("domain", domain).arg("from_key", fromKey) + .arg("to_key", toKey).build(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + String message = response.getMessage(); + handleUnknownKeyException(fromKey, domain, message); + handleKeyAlreadyExists(domain, toKey, message); + throw new TrackerException(message); + } + } + + @Override + public void updateStorageClass(String key, String domain, String newStorageClass) throws TrackerException { + Request request = new Request.Builder(3).command("updateclass").arg("domain", domain).arg("key", key) + .arg("class", newStorageClass).build(); + Response response = requestHandler.performRequest(request); + if (response.getStatus() != OK) { + String message = response.getMessage(); + handleUnknownKeyException(key, domain, message); + handleUnknownStorageClass(newStorageClass, message); + throw new TrackerException(message); + } + } + + @Override + public void noop() throws TrackerException { + Request request = new Request.Builder(0).command("noop").build(); + Response response = requestHandler.performRequest(request); + handleGeneralResponseError(response); + } + + @Override + public void close() { + if (requestHandler != null) { + requestHandler.close(); + } + IOUtils.closeQuietly(socket); + log.debug("Closed"); + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("TrackerImpl [socket="); + builder.append(socket); + builder.append(", requestHandler="); + builder.append(requestHandler); + builder.append("]"); + return builder.toString(); + } + + private void handleUnknownStorageClass(String storageClass, String message) throws UnknownStorageClassException { + if (UNKNOWN_CLASS.isContainedInLine(message)) { + throw new UnknownStorageClassException(storageClass); + } + } + + private void handleKeyAlreadyExists(String domain, String key, String message) throws KeyExistsAlreadyException { + if (KEY_EXISTS.isContainedInLine(message)) { + throw new KeyExistsAlreadyException(domain, key); + } + } + + private void handleUnknownKeyException(String key, String domain, String message) throws UnknownKeyException { + if (UNKNOWN_KEY.isContainedInLine(message)) { + throw new UnknownKeyException(domain, key); + } + } + + private void handleGeneralResponseError(Response response) throws TrackerException { + if (response.getStatus() != OK) { + throw new TrackerException(response.getMessage()); + } + } + +} diff --git a/src/main/java/fm/last/moji/tracker/pool/BorrowedTracker.java b/src/main/java/fm/last/moji/tracker/pool/BorrowedTracker.java new file mode 100644 index 0000000..5a54840 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/pool/BorrowedTracker.java @@ -0,0 +1,187 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.pool; + +import java.net.URL; +import java.util.Collections; +import java.util.List; + +import org.apache.commons.pool.KeyedObjectPool; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.impl.CommunicationException; + +class BorrowedTracker implements Tracker { + + private static final Logger log = LoggerFactory.getLogger(BorrowedTracker.class); + + private final Tracker delegate; + private final KeyedObjectPool pool; + private final ManagedTrackerHost host; + private CommunicationException lastException; + + BorrowedTracker(ManagedTrackerHost host, Tracker delegate, KeyedObjectPool pool) { + this.delegate = delegate; + this.host = host; + this.pool = pool; + } + + @Override + public List getPaths(String key, String domain) throws TrackerException { + List paths = Collections.emptyList(); + try { + paths = delegate.getPaths(key, domain); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + return paths; + } + + @Override + public List createOpen(String key, String domain, String storageClass) throws TrackerException { + List destinations = Collections.emptyList(); + try { + destinations = delegate.createOpen(key, domain, storageClass); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + return destinations; + } + + @Override + public void createClose(String key, String domain, Destination destination, long size) throws TrackerException { + try { + delegate.createClose(key, domain, destination, size); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + } + + @Override + public void delete(String key, String domain) throws TrackerException { + try { + delegate.delete(key, domain); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + } + + @Override + public void rename(String key, String domain, String newKey) throws TrackerException { + try { + delegate.rename(key, domain, newKey); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + } + + @Override + public void updateStorageClass(String key, String domain, String newStorageClass) throws TrackerException { + try { + delegate.updateStorageClass(key, domain, newStorageClass); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + } + + @Override + public void noop() throws TrackerException { + try { + delegate.noop(); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + } + + @Override + public List list(String domain, String keyPrefix, Integer limit) throws TrackerException { + List keys = Collections.emptyList(); + try { + keys = delegate.list(domain, keyPrefix, limit); + host.markSuccess(); + } catch (CommunicationException e) { + lastException = e; + throw e; + } + return keys; + } + + @Override + public void close() { + try { + if (lastException != null) { + log.debug("Invalidating: {}", lastException); + try { + pool.invalidateObject(host, this); + } finally { + delegate.close(); + } + } else { + log.debug("Returning to pool"); + pool.returnObject(host, this); + } + } catch (Exception e) { + throw new RuntimeException(e); + } + } + + void reallyClose() { + log.debug("Really closing"); + delegate.close(); + } + + CommunicationException getLastException() { + return lastException; + } + + ManagedTrackerHost getHost() { + return host; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("BorrowedTracker [host="); + builder.append(host); + builder.append(", lastException="); + builder.append(lastException); + builder.append(", delegate="); + builder.append(delegate); + builder.append(", pool="); + builder.append(pool); + builder.append("]"); + return builder.toString(); + } + +} diff --git a/src/main/java/fm/last/moji/tracker/pool/HostPriorityComparator.java b/src/main/java/fm/last/moji/tracker/pool/HostPriorityComparator.java new file mode 100644 index 0000000..af408a3 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/pool/HostPriorityComparator.java @@ -0,0 +1,51 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.pool; + +import java.io.Serializable; +import java.util.Comparator; + +class HostPriorityComparator implements Comparator, Serializable { + + private static final long serialVersionUID = 1L; + + @Override + public int compare(ManagedTrackerHost a, ManagedTrackerHost b) { + long failTime1 = a.getLastFailed(); + long failTime2 = b.getLastFailed(); + if (failTime1 == failTime2) { + // they both failed at the same time (or not at all) so we just want to + // priotitise the least recently used. + long useTime1 = a.getLastUsed(); + long useTime2 = b.getLastUsed(); + if (useTime1 == useTime2) { + return 0; + } else if (useTime1 > useTime2) { + // 'a' was used more recently than 'b' - reduce the priority and + // chose 'b' before it + return -1; + } + return 1; + } else if (failTime1 < failTime2) { + // 'b' failed more recently - so we prioritise 'a' in the hopes that it + // has had longer to recover + return 1; + } + // 'a' failed more recently - reduce it's priority + return -1; + } + +} diff --git a/src/main/java/fm/last/moji/tracker/pool/ManagedTrackerHost.java b/src/main/java/fm/last/moji/tracker/pool/ManagedTrackerHost.java new file mode 100644 index 0000000..5bc2a7c --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/pool/ManagedTrackerHost.java @@ -0,0 +1,155 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.pool; + +import java.net.InetSocketAddress; +import java.util.Date; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.atomic.AtomicLong; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Manages the status of a given tracker host. Knows when it last failed, when the last successful request occurred. + */ +public class ManagedTrackerHost { + + private static final Logger log = LoggerFactory.getLogger(ManagedTrackerHost.class); + + private static final int ONE_MINUTE_IN_MS = 60000; + + private final AtomicLong lastUsed = new AtomicLong(); + private final AtomicLong lastFailed = new AtomicLong(); + private final InetSocketAddress address; + private Timer resetTimer; + private ResetTask resetTask; + + ManagedTrackerHost(InetSocketAddress address) { + this.address = address; + } + + /** + * The address of the host that this object manages. + * + * @return Host address + */ + public InetSocketAddress getAddress() { + lastUsed.set(System.currentTimeMillis()); + return address; + } + + /** + * The time when this host was last used. + * + * @return Date/Time formatted string + */ + public String getLastUsedTime() { + return formatTime(getLastUsed()); + } + + /** + * The time when an operation on this host last failed. + * + * @return Date/Time formatted string + */ + public String getLastFailedTime() { + return formatTime(getLastFailed()); + } + + long getLastUsed() { + return lastUsed.get(); + } + + long getLastFailed() { + return lastFailed.get(); + } + + void markAsFailed() { + lastFailed.set(System.currentTimeMillis()); + synchronized (resetTimer) { + if (resetTask != null) { + resetTask.cancel(); + } + log.debug("Scheduling reset of {} in {}ms", address, ONE_MINUTE_IN_MS); + resetTimer.schedule(new ResetTask(), ONE_MINUTE_IN_MS); + } + } + + void markSuccess() { + lastFailed.set(0); + } + + @Override + public int hashCode() { + final int prime = 31; + int result = 1; + result = prime * result + (address == null ? 0 : address.hashCode()); + return result; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (!(obj instanceof ManagedTrackerHost)) { + return false; + } + ManagedTrackerHost other = (ManagedTrackerHost) obj; + if (address == null) { + if (other.address != null) { + return false; + } + } else if (!address.equals(other.address)) { + return false; + } + return true; + } + + @Override + public String toString() { + StringBuilder builder = new StringBuilder(); + builder.append("ManagedTrackerAddress [address="); + builder.append(address); + builder.append(", lastUsed="); + builder.append(formatTime(lastUsed.get())); + builder.append(", lastFailed="); + builder.append(formatTime(lastFailed.get())); + builder.append("]"); + return builder.toString(); + } + + private String formatTime(long time) { + if (time == 0L) { + return ""; + } + return new Date(time).toString(); + } + + private final class ResetTask extends TimerTask { + @Override + public void run() { + lastFailed.set(0); + log.debug("Reset failure monitor for {}", address); + } + } + +} diff --git a/src/main/java/fm/last/moji/tracker/pool/MultiHostTrackerPool.java b/src/main/java/fm/last/moji/tracker/pool/MultiHostTrackerPool.java new file mode 100644 index 0000000..def01c1 --- /dev/null +++ b/src/main/java/fm/last/moji/tracker/pool/MultiHostTrackerPool.java @@ -0,0 +1,246 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.pool; + +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.commons.pool.BaseKeyedPoolableObjectFactory; +import org.apache.commons.pool.KeyedPoolableObjectFactory; +import org.apache.commons.pool.impl.GenericKeyedObjectPool; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.TrackerFactory; +import fm.last.moji.tracker.impl.AbstractTrackerFactory; + +/** + * {@link fm.last.moji.tracker.TrackerFactory TrackerFactory} implementation that provides a + * {@link fm.last.moji.tracker.Tracker Tracker} connection pool that can distribute requests across many hosts. + */ +public class MultiHostTrackerPool implements TrackerFactory { + + private static final Logger log = LoggerFactory.getLogger(MultiHostTrackerPool.class); + private static final HostPriorityComparator PRIORITY_COMPARATOR = new HostPriorityComparator(); + + private final Proxy proxy; + private final GenericKeyedObjectPool pool; + private final List managedHosts; + + /** + * Creates a tracker pool for the given host addresses and use the supplied network proxy. + * + * @param addresses Tracker host addresses + * @param proxy Network proxy - use Proxy.NO_PROXY if a proxy isn't needed. + */ + public MultiHostTrackerPool(Set addresses, Proxy proxy) { + this.proxy = proxy; + managedHosts = new ArrayList(); + for (InetSocketAddress address : addresses) { + managedHosts.add(new ManagedTrackerHost(address)); + } + KeyedPoolableObjectFactory objectFactory = new BorrowedTrackerObjectPoolFactory(); + pool = new GenericKeyedObjectPool(objectFactory); + log.debug("Pool created"); + } + + @Override + public Tracker getTracker() throws TrackerException { + ManagedTrackerHost managedHost = nextHost(); + Tracker tracker = null; + try { + tracker = (Tracker) pool.borrowObject(managedHost); + } catch (Exception e) { + throw new TrackerException(e); + } + return tracker; + } + + @Override + public Set getAddresses() { + Set addresses = new HashSet(); + for (ManagedTrackerHost host : managedHosts) { + addresses.add(host.getAddress()); + } + return Collections.unmodifiableSet(new HashSet(addresses)); + } + + @Override + public Proxy getProxy() { + return proxy; + } + + /** + * Returns the status of the hosts managed by this pool. + * + * @return A list of hosts statuses. + */ + public List getManagedHosts() { + return managedHosts; + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setMaxActive(int)} + */ + public void setMaxActive(int maxActive) { + pool.setMaxActive(maxActive); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxWait()} + */ + public long getMaxWait() { + return pool.getMaxWait(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setMaxWait(long)} + */ + public void setMaxWait(long maxWait) { + pool.setMaxWait(maxWait); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxIdle()} + */ + public void setMaxIdle(int maxIdle) { + pool.setMaxIdle(maxIdle); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getTestOnBorrow()} + */ + public boolean getTestOnBorrow() { + return pool.getTestOnBorrow(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setTestOnBorrow(boolean)} + */ + public void setTestOnBorrow(boolean testOnBorrow) { + pool.setTestOnBorrow(testOnBorrow); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getTestOnReturn()} + */ + public boolean getTestOnReturn() { + return pool.getTestOnReturn(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#setTestOnReturn(boolean)} + */ + public void setTestOnReturn(boolean testOnReturn) { + pool.setTestOnReturn(testOnReturn); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getNumActive()} + */ + public int getNumActive() { + return pool.getNumActive(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getNumIdle()} + */ + public int getNumIdle() { + return pool.getNumIdle(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxActive()} + */ + public int getMaxActive() { + return pool.getMaxActive(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#getMaxIdle()} + */ + public int getMaxIdle() { + return pool.getMaxIdle(); + } + + /** + * See: {@link org.apache.commons.pool.impl.GenericKeyedObjectPool#close()} + */ + public void close() throws Exception { + pool.close(); + } + + private ManagedTrackerHost nextHost() throws TrackerException { + ManagedTrackerHost managedHost = null; + synchronized (managedHosts) { + Collections.sort(managedHosts, PRIORITY_COMPARATOR); + managedHost = managedHosts.get(managedHosts.size() - 1); + } + return managedHost; + } + + private class BorrowedTrackerObjectPoolFactory extends BaseKeyedPoolableObjectFactory { + + private final AbstractTrackerFactory delegateFactory; + + BorrowedTrackerObjectPoolFactory() { + delegateFactory = new AbstractTrackerFactory(proxy); + } + + @Override + public Object makeObject(Object key) throws Exception { + ManagedTrackerHost host = (ManagedTrackerHost) key; + Tracker delegate = delegateFactory.newTracker(host.getAddress()); + BorrowedTracker borrowedTracker = new BorrowedTracker(host, delegate, pool); + log.debug("Requested new tracker instance: {}", key); + return borrowedTracker; + } + + @Override + public void destroyObject(Object key, Object obj) throws Exception { + BorrowedTracker borrowed = (BorrowedTracker) obj; + if (borrowed.getLastException() != null) { + log.debug("Error occurred on tracker: {}", borrowed.getLastException().getMessage()); + borrowed.getHost().markAsFailed(); + } + log.debug("Destroying {}", borrowed); + borrowed.reallyClose(); + } + + @Override + public boolean validateObject(Object key, Object obj) { + BorrowedTracker borrowed = (BorrowedTracker) obj; + log.debug("Validating {}", borrowed); + try { + borrowed.noop(); + } catch (TrackerException e) { + // returning false will result in a destroyObject invocation + // The address will then be marked out of service + return false; + } + return true; + } + + } + +} diff --git a/src/main/resources/spring/moji-context.xml b/src/main/resources/spring/moji-context.xml new file mode 100644 index 0000000..2f25d06 --- /dev/null +++ b/src/main/resources/spring/moji-context.xml @@ -0,0 +1,18 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/src/test/data/fileOfKnownSize.dat b/src/test/data/fileOfKnownSize.dat new file mode 100644 index 0000000..7d1f19e Binary files /dev/null and b/src/test/data/fileOfKnownSize.dat differ diff --git a/src/test/data/mogileFileCopyToFile.dat b/src/test/data/mogileFileCopyToFile.dat new file mode 100644 index 0000000..ac0b5bc Binary files /dev/null and b/src/test/data/mogileFileCopyToFile.dat differ diff --git a/src/test/java/fm/last/moji/FakeMogileFsServer.java b/src/test/java/fm/last/moji/FakeMogileFsServer.java new file mode 100644 index 0000000..3e4fae0 --- /dev/null +++ b/src/test/java/fm/last/moji/FakeMogileFsServer.java @@ -0,0 +1,154 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.commons.io.IOUtils; +import org.junit.Ignore; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +@Ignore +public class FakeMogileFsServer { + + private static final Logger log = LoggerFactory.getLogger(FakeMogileFsServer.class); + + private ServerSocket trackerSocket; + private Thread trackerThread; + + FakeMogileFsServer(final Builder builder) throws Exception { + startTracker(builder); + } + + public String getAddressAsString() { + return trackerSocket.getInetAddress().getHostName() + ":" + trackerSocket.getLocalPort(); + } + + public InetAddress getInetAddress() { + return trackerSocket.getInetAddress(); + } + + public InetSocketAddress getInetSocketAddress() { + return new InetSocketAddress(trackerSocket.getInetAddress().getHostName(), trackerSocket.getLocalPort()); + } + + public int getPort() { + return trackerSocket.getLocalPort(); + } + + public void close() throws Exception { + log.info("Closing"); + try { + trackerThread.interrupt(); + } finally { + trackerSocket.close(); + } + } + + private void startTracker(final Builder builder) throws IOException { + trackerSocket = new ServerSocket(0); + trackerThread = new TrackerServer(builder.conversation); + trackerThread.start(); + log.info("Tracker server running on: {}", getAddressAsString()); + } + + private final class TrackerServer extends Thread { + private final List conversation; + + private TrackerServer(List conversation) { + this.conversation = conversation; + } + + @Override + public void run() { + Socket accept = null; + BufferedReader reader = null; + OutputStreamWriter writer = null; + try { + accept = trackerSocket.accept(); + writer = new OutputStreamWriter(accept.getOutputStream()); + reader = new BufferedReader(new InputStreamReader(accept.getInputStream())); + + for (Stanza stanza : conversation) { + String line = reader.readLine(); + for (String element : stanza.responses) { + if (!line.contains(element)) { + return; + } + } + writer.write(stanza.request); + writer.flush(); + } + } catch (IOException e) { + e.printStackTrace(); + } finally { + IOUtils.closeQuietly(reader); + IOUtils.closeQuietly(writer); + IOUtils.closeQuietly(accept); + } + } + } + + public static class Builder { + + private final List conversation; + private Stanza current; + + public Builder() { + conversation = new ArrayList(); + } + + public Builder whenRequestContains(String... strings) { + current = new Stanza(); + current.responses = new HashSet(Arrays.asList(strings)); + return this; + } + + public Builder thenRespond(String string) { + current.request = string; + conversation.add(current); + current = null; + return this; + } + + public FakeMogileFsServer build() throws Exception { + return new FakeMogileFsServer(this); + } + + } + + private static class Stanza { + private String request; + private Set responses; + + private Stanza() { + } + } + +} diff --git a/src/test/java/fm/last/moji/MojiInstantiationTest.java b/src/test/java/fm/last/moji/MojiInstantiationTest.java new file mode 100644 index 0000000..d3cc2dd --- /dev/null +++ b/src/test/java/fm/last/moji/MojiInstantiationTest.java @@ -0,0 +1,45 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji; + +import java.net.Proxy; + +import org.junit.Test; + +import fm.last.moji.impl.DefaultMojiFactory; +import fm.last.moji.tracker.TrackerFactory; +import fm.last.moji.tracker.impl.SingleHostTrackerFactory; + +public class MojiInstantiationTest { + + @Test(timeout = 2000) + public void delete() throws Exception { + FakeMogileFsServer server = null; + try { + FakeMogileFsServer.Builder builder = new FakeMogileFsServer.Builder(); + builder.whenRequestContains("delete ", "key=myKey", "domain=myDomain").thenRespond("OK "); + server = builder.build(); + TrackerFactory trackerFactory = new SingleHostTrackerFactory(server.getInetSocketAddress(), Proxy.NO_PROXY); + DefaultMojiFactory mojiFactory = new DefaultMojiFactory(trackerFactory, "myDomain"); + Moji moji = mojiFactory.getInstance(); + MojiFile file = moji.getFile("myKey"); + file.delete(); + } finally { + server.close(); + } + } + +} diff --git a/src/test/java/fm/last/moji/impl/DeleteCommandTest.java b/src/test/java/fm/last/moji/impl/DeleteCommandTest.java new file mode 100644 index 0000000..4c4e27b --- /dev/null +++ b/src/test/java/fm/last/moji/impl/DeleteCommandTest.java @@ -0,0 +1,46 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.verify; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Tracker; + +@RunWith(MockitoJUnitRunner.class) +public class DeleteCommandTest { + + @Mock + private Tracker mockTracker; + private DeleteCommand command; + + @Before + public void init() { + command = new DeleteCommand("key", "domain"); + } + + @Test + public void delegatesToTracker() throws Exception { + command.executeWithTracker(mockTracker); + verify(mockTracker).delete("key", "domain"); + } + +} diff --git a/src/test/java/fm/last/moji/impl/ExecutorTest.java b/src/test/java/fm/last/moji/impl/ExecutorTest.java new file mode 100644 index 0000000..b836be0 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/ExecutorTest.java @@ -0,0 +1,84 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.util.Collections; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.TrackerFactory; +import fm.last.moji.tracker.UnknownKeyException; + +@RunWith(MockitoJUnitRunner.class) +public class ExecutorTest { + + @Mock + private Tracker mockTracker; + @Mock + private TrackerFactory mockFactory; + @Mock + private MojiCommand mockCommand; + @Mock + private InetSocketAddress mockAddress; + + private Executor executor; + + @Before + public void init() throws TrackerException { + when(mockFactory.getTracker()).thenReturn(mockTracker); + when(mockFactory.getAddresses()).thenReturn(Collections.singleton(mockAddress)); + executor = new Executor(mockFactory); + } + + @Test + public void executeCommandOk() throws Exception { + executor.executeCommand(mockCommand); + verify(mockFactory).getTracker(); + verify(mockCommand).executeWithTracker(mockTracker); + verify(mockTracker).close(); + } + + @Test(expected = UnknownKeyException.class) + public void trackerClosedOnUnknownKeyException() throws Exception { + UnknownKeyException e = new UnknownKeyException("domain", "key"); + doThrow(e).when(mockCommand).executeWithTracker(mockTracker); + + executor.executeCommand(mockCommand); + verify(mockTracker).close(); + } + + @Test(expected = IOException.class) + public void trackerClosedOnIOException() throws Exception { + IOException e = new IOException(); + doThrow(e).when(mockCommand).executeWithTracker(mockTracker); + + executor.executeCommand(mockCommand); + verify(mockTracker).close(); + } + +} diff --git a/src/test/java/fm/last/moji/impl/ExistsCommandTest.java b/src/test/java/fm/last/moji/impl/ExistsCommandTest.java new file mode 100644 index 0000000..3476e75 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/ExistsCommandTest.java @@ -0,0 +1,71 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.when; + +import java.net.URL; +import java.util.Collections; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.UnknownKeyException; + +@RunWith(MockitoJUnitRunner.class) +public class ExistsCommandTest { + + @Mock + private Tracker mockTracker; + private ExistsCommand command; + + @Before + public void init() { + command = new ExistsCommand("key", "domain"); + } + + @Test + public void onePath() throws Exception { + List paths = Collections.singletonList(new URL("http://www.last.fm")); + when(mockTracker.getPaths("key", "domain")).thenReturn(paths); + command.executeWithTracker(mockTracker); + assertTrue(command.getExists()); + } + + @Test + public void zeroPaths() throws Exception { + List paths = Collections.emptyList(); + when(mockTracker.getPaths("key", "domain")).thenReturn(paths); + command.executeWithTracker(mockTracker); + assertFalse(command.getExists()); + } + + @Test + public void unknownKeyException() throws Exception { + UnknownKeyException e = new UnknownKeyException("key", "domain"); + when(mockTracker.getPaths("key", "domain")).thenThrow(e); + command.executeWithTracker(mockTracker); + assertFalse(command.getExists()); + } + +} diff --git a/src/test/java/fm/last/moji/impl/FileDownloadInputStreamTest.java b/src/test/java/fm/last/moji/impl/FileDownloadInputStreamTest.java new file mode 100644 index 0000000..9027f25 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/FileDownloadInputStreamTest.java @@ -0,0 +1,112 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; + +import java.io.IOException; +import java.io.InputStream; +import java.util.concurrent.locks.Lock; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +@RunWith(MockitoJUnitRunner.class) +public class FileDownloadInputStreamTest { + + @Mock + private InputStream mockInputStream; + @Mock + private Lock mockReadLock; + private FileDownloadInputStream stream; + + @Before + public void setUp() { + stream = new FileDownloadInputStream(mockInputStream, mockReadLock); + } + + @Test + public void readDelegates() throws IOException { + stream.read(); + verify(mockInputStream).read(); + } + + @Test + public void readByteArrayDelegates() throws IOException { + byte[] b = new byte[4]; + stream.read(b); + verify(mockInputStream).read(b); + } + + @Test + public void readByteArrayWithOffsetDelegates() throws IOException { + byte[] b = new byte[4]; + stream.read(b, 2, 4); + verify(mockInputStream).read(b, 2, 4); + } + + @Test + public void skipDelegates() throws IOException { + stream.skip(1); + verify(mockInputStream).skip(1); + } + + @Test + public void availableDelegates() throws IOException { + stream.available(); + verify(mockInputStream).available(); + } + + @Test + public void closeDelegates() throws IOException { + stream.close(); + verify(mockInputStream).close(); + verify(mockReadLock).unlock(); + } + + @Test + public void closeReleasesLockEvenOnError() throws IOException { + doThrow(new IOException()).when(mockInputStream).close(); + try { + stream.close(); + } catch (Exception e) { + } + verify(mockReadLock).unlock(); + } + + @Test + public void markDelegates() { + stream.mark(21); + verify(mockInputStream).mark(21); + } + + @Test + public void resetDelegates() throws IOException { + stream.reset(); + verify(mockInputStream).reset(); + } + + @Test + public void markSupportedDelegates() { + stream.markSupported(); + verify(mockInputStream).markSupported(); + } + +} diff --git a/src/test/java/fm/last/moji/impl/FileUploadOutputStreamTest.java b/src/test/java/fm/last/moji/impl/FileUploadOutputStreamTest.java new file mode 100644 index 0000000..4f0aa8c --- /dev/null +++ b/src/test/java/fm/last/moji/impl/FileUploadOutputStreamTest.java @@ -0,0 +1,155 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.HttpURLConnection; +import java.net.URL; +import java.util.concurrent.locks.Lock; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerFactory; + +@RunWith(MockitoJUnitRunner.class) +public class FileUploadOutputStreamTest { + + private static final String KEY = "key"; + private static final String DOMAIN = "domain"; + @Mock + private TrackerFactory mockTrackerFactory; + @Mock + private HttpConnectionFactory mockHttpFactory; + @Mock + private Destination mockDestination; + @Mock + private HttpURLConnection mockHttpConnection; + @Mock + private OutputStream mockOutputStream; + @Mock + private InputStream mockInputStream; + @Mock + private Tracker mockTracker; + @Mock + private Lock mockWriteLock; + private FileUploadOutputStream stream; + + @Before + public void setUp() throws IOException { + URL url = new URL("http://www.last.fm/"); + when(mockDestination.getPath()).thenReturn(url); + when(mockHttpFactory.newConnection(url)).thenReturn(mockHttpConnection); + when(mockHttpConnection.getInputStream()).thenReturn(mockInputStream); + when(mockHttpConnection.getOutputStream()).thenReturn(mockOutputStream); + when(mockHttpConnection.getResponseMessage()).thenReturn("message"); + when(mockHttpConnection.getResponseCode()).thenReturn(HttpURLConnection.HTTP_OK); + when(mockTrackerFactory.getTracker()).thenReturn(mockTracker); + + stream = new FileUploadOutputStream(mockTrackerFactory, mockHttpFactory, KEY, DOMAIN, mockDestination, + mockWriteLock); + } + + @Test + public void httpConnectionSetUp() throws IOException { + verify(mockHttpConnection).setRequestMethod("PUT"); + verify(mockHttpConnection).setChunkedStreamingMode(4096); + verify(mockHttpConnection).setDoOutput(true); + } + + @Test + public void everyThingCloses() throws IOException { + stream.write(1); + stream.close(); + + verify(mockOutputStream).flush(); + verify(mockOutputStream).close(); + verify(mockHttpConnection).disconnect(); + verify(mockTracker).createClose(KEY, DOMAIN, mockDestination, 1); + verify(mockTracker).close(); + verify(mockWriteLock).unlock(); + } + + @Test + public void everyThingClosesEvenOnFail() throws IOException { + doThrow(new RuntimeException()).when(mockOutputStream).flush(); + doThrow(new RuntimeException()).when(mockOutputStream).close(); + when(mockHttpConnection.getResponseCode()).thenReturn(HttpURLConnection.HTTP_INTERNAL_ERROR); + doThrow(new RuntimeException()).when(mockInputStream).close(); + doThrow(new RuntimeException()).when(mockHttpConnection).disconnect(); + doThrow(new RuntimeException()).when(mockTracker).createClose(KEY, DOMAIN, mockDestination, 1); + + try { + stream.write(1); + stream.close(); + } catch (Exception e) { + } + + verify(mockOutputStream).flush(); + verify(mockOutputStream).close(); + verify(mockHttpConnection).disconnect(); + verify(mockTracker).createClose(KEY, DOMAIN, mockDestination, -1); + verify(mockTracker).close(); + verify(mockWriteLock).unlock(); + } + + @Test + public void writeIntDelegates() throws IOException { + stream.write(1); + verify(mockOutputStream).write(1); + } + + @Test + public void writeByteArrayDelegates() throws IOException { + byte[] b = new byte[4]; + stream.write(b); + verify(mockOutputStream).write(b); + } + + @Test + public void writeByteArrayWithOffsetDelegates() throws IOException { + byte[] b = new byte[4]; + stream.write(b, 2, 4); + verify(mockOutputStream).write(b, 2, 4); + } + + @Test + public void flushDelegates() throws IOException { + stream.flush(); + verify(mockOutputStream).flush(); + } + + @Test + public void countOnWrite() throws IOException { + byte[] b = new byte[] { 1, 2, 3, 4, 5 }; + stream.write(b); + stream.flush(); + stream.close(); + verify(mockTracker).createClose(KEY, DOMAIN, mockDestination, 5); + } + +} diff --git a/src/test/java/fm/last/moji/impl/HttpConnectionFactoryTest.java b/src/test/java/fm/last/moji/impl/HttpConnectionFactoryTest.java new file mode 100644 index 0000000..bb613bc --- /dev/null +++ b/src/test/java/fm/last/moji/impl/HttpConnectionFactoryTest.java @@ -0,0 +1,63 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.CoreMatchers.not; +import static org.hamcrest.CoreMatchers.nullValue; +import static org.junit.Assert.assertThat; + +import java.io.IOException; +import java.net.HttpURLConnection; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.net.ServerSocket; +import java.net.URL; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +public class HttpConnectionFactoryTest { + + private HttpConnectionFactory factory; + private ServerSocket serverSocket; + private InetSocketAddress address; + private HttpURLConnection connection; + + @Before + public void init() throws IOException { + serverSocket = new ServerSocket(0); + address = new InetSocketAddress(serverSocket.getInetAddress(), serverSocket.getLocalPort()); + factory = new HttpConnectionFactory(Proxy.NO_PROXY); + } + + @After + public void tearDown() throws IOException { + try { + connection.disconnect(); + } finally { + serverSocket.close(); + } + } + + @Test + public void newConnection() throws Exception { + connection = factory.newConnection(new URL("http://" + address.getHostName() + ":" + address.getPort() + "/")); + assertThat(connection, is(not(nullValue()))); + } + +} diff --git a/src/test/java/fm/last/moji/impl/ListFilesCommandTest.java b/src/test/java/fm/last/moji/impl/ListFilesCommandTest.java new file mode 100644 index 0000000..64b7971 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/ListFilesCommandTest.java @@ -0,0 +1,89 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.runners.MockitoJUnitRunner; +import org.mockito.stubbing.Answer; + +import fm.last.moji.Moji; +import fm.last.moji.MojiFile; +import fm.last.moji.tracker.Tracker; + +@RunWith(MockitoJUnitRunner.class) +public class ListFilesCommandTest { + + private static final String DOMAIN = "domain"; + private static final String KEY_PREFIX = "key"; + + @Mock + private Moji mockMoji; + @Mock + private Tracker mockTracker; + + @Before + public void setUp() { + when(mockMoji.getFile(anyString())).thenAnswer(new Answer() { + @Override + public MojiFile answer(InvocationOnMock invocation) throws Throwable { + Object[] arguments = invocation.getArguments(); + MojiFile mock = mock(MojiFile.class); + when(mock.getKey()).thenReturn((String) arguments[0]); + return mock; + } + }); + } + + @Test + public void list() throws IOException { + List keys = Arrays.asList(new String[] { "key1", "key2", "key3" }); + when(mockTracker.list(DOMAIN, KEY_PREFIX, null)).thenReturn(keys); + ListFilesCommand command = new ListFilesCommand(mockMoji, KEY_PREFIX, DOMAIN); + command.executeWithTracker(mockTracker); + List fileList = command.getFileList(); + assertThat(fileList.size(), is(3)); + assertThat(fileList.get(0).getKey(), is("key1")); + assertThat(fileList.get(1).getKey(), is("key2")); + assertThat(fileList.get(2).getKey(), is("key3")); + } + + @Test + public void listWithLimit() throws IOException { + List keys = Collections.singletonList("key1"); + when(mockTracker.list(DOMAIN, KEY_PREFIX, 1)).thenReturn(keys); + ListFilesCommand command = new ListFilesCommand(mockMoji, KEY_PREFIX, DOMAIN, 1); + command.executeWithTracker(mockTracker); + List fileList = command.getFileList(); + assertThat(fileList.size(), is(1)); + assertThat(fileList.get(0).getKey(), is("key1")); + } + +} diff --git a/src/test/java/fm/last/moji/impl/MojiFileImplTest.java b/src/test/java/fm/last/moji/impl/MojiFileImplTest.java new file mode 100644 index 0000000..aa07738 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/MojiFileImplTest.java @@ -0,0 +1,254 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; +import static org.mockito.Matchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.HttpURLConnection; +import java.net.InetSocketAddress; +import java.net.URL; +import java.util.Collections; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.TrackerFactory; + +@RunWith(MockitoJUnitRunner.class) +public class MojiFileImplTest { + + private static final String DOMAIN = "domain"; + private static final String STORAGE_CLASS = "storageClass"; + private static final String STORAGE_CLASS_2 = "newStorageClass"; + private static final String KEY = "key"; + private static final String KEY_2 = "newKey"; + + @Mock + private Tracker mockTracker; + @Mock + private TrackerFactory mockTrackerFactory; + @Mock + private HttpConnectionFactory mockHttpFactory; + @Mock + private Executor mockExecutor; + @Mock + private HttpURLConnection mockUrlConnection; + @Mock + private InputStream mockInputStream; + @Mock + private OutputStream mockOutputStream; + @Mock + private InetSocketAddress mockAddress; + + private MojiFileImpl file; + + @Before + public void init() throws TrackerException { + when(mockTrackerFactory.getTracker()).thenReturn(mockTracker); + when(mockTrackerFactory.getAddresses()).thenReturn(Collections.singleton(mockAddress)); + file = new MojiFileImpl(KEY, DOMAIN, STORAGE_CLASS, mockTrackerFactory, mockHttpFactory); + } + + @Test + public void existsCommand() throws IOException { + file.setExecutor(mockExecutor); + file.exists(); + ArgumentCaptor captor = ArgumentCaptor.forClass(ExistsCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + } + + @Test + public void existsReturn() throws IOException { + List paths = Collections.singletonList(new URL("http://localhost:80/")); + when(mockTracker.getPaths(KEY, DOMAIN)).thenReturn(paths); + boolean exists = file.exists(); + assertThat(exists, is(true)); + } + + @Test + public void deleteCommand() throws IOException { + file.setExecutor(mockExecutor); + file.delete(); + ArgumentCaptor captor = ArgumentCaptor.forClass(DeleteCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + } + + @Test + public void renameCommand() throws IOException { + file.setExecutor(mockExecutor); + file.rename(KEY_2); + ArgumentCaptor captor = ArgumentCaptor.forClass(RenameCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + assertThat(captor.getValue().newKey, is(KEY_2)); + } + + @Test + public void renameChangesKeyInFile() throws IOException { + assertThat(file.getKey(), is(KEY)); + file.rename(KEY_2); + assertThat(file.getKey(), is(KEY_2)); + } + + @Test + public void renameKeyNotModifiedOnError() throws IOException { + file.setExecutor(mockExecutor); + doThrow(new IOException()).when(mockExecutor).executeCommand(any(RenameCommand.class)); + assertThat(file.getKey(), is(KEY)); + try { + file.rename(KEY_2); + } catch (IOException ignored) { + } + assertThat(file.getKey(), is(KEY)); + } + + @Test + public void storageClassCommand() throws IOException { + file.setExecutor(mockExecutor); + file.modifyStorageClass(STORAGE_CLASS_2); + ArgumentCaptor captor = ArgumentCaptor.forClass(UpdateStorageClassCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + assertThat(captor.getValue().newStorageClass, is(STORAGE_CLASS_2)); + } + + // We cannot know the storage class currently + // @Test + // public void storageClassChangesInFile() throws IOException { + // assertThat(file.getStorageClass(), is(STORAGE_CLASS)); + // file.modifyStorageClass(STORAGE_CLASS_2); + // assertThat(file.getStorageClass(), is(STORAGE_CLASS_2)); + // } + + // We cannot know the storage class currently + // @Test + // public void storageClassNotModifiedOnError() throws IOException { + // file.setExecutor(mockExecutor); + // doThrow(new IOException()).when(mockExecutor).executeCommand(any(UpdateStorageClassCommand.class)); + // assertThat(file.getStorageClass(), is(STORAGE_CLASS)); + // try { + // file.modifyStorageClass(STORAGE_CLASS_2); + // } catch (IOException ignored) { + // } + // assertThat(file.getStorageClass(), is(STORAGE_CLASS)); + // } + + @Test + public void lengthCommand() throws IOException { + file.setExecutor(mockExecutor); + file.length(); + ArgumentCaptor captor = ArgumentCaptor.forClass(FileLengthCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + } + + @Test + public void lengthReturn() throws IOException { + URL path = new URL("http://localhost:80/"); + List paths = Collections.singletonList(path); + when(mockTracker.getPaths(KEY, DOMAIN)).thenReturn(paths); + when(mockHttpFactory.newConnection(path)).thenReturn(mockUrlConnection); + when(mockUrlConnection.getContentLength()).thenReturn(74634654); + + // check that whatever we have delegates to the expected stream + long length = file.length(); + assertThat(length, is(74634654L)); + } + + @Test + public void getInputStreamCommand() throws IOException { + file.setExecutor(mockExecutor); + file.getInputStream(); + ArgumentCaptor captor = ArgumentCaptor.forClass(GetInputStreamCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + } + + @Test + public void getInputStreamCommandReturn() throws IOException { + URL path = new URL("http://localhost:80/"); + List paths = Collections.singletonList(path); + when(mockTracker.getPaths(KEY, DOMAIN)).thenReturn(paths); + when(mockHttpFactory.newConnection(path)).thenReturn(mockUrlConnection); + when(mockUrlConnection.getInputStream()).thenReturn(mockInputStream); + + // check that whatever we have delegates to the expected stream + InputStream inputStream = file.getInputStream(); + byte[] myBuffer = new byte[2]; + inputStream.read(myBuffer, 2, 43); + verify(mockInputStream).read(myBuffer, 2, 43); + } + + @Test + public void getOutputStreamCommand() throws IOException { + file.setExecutor(mockExecutor); + file.getOutputStream(); + ArgumentCaptor captor = ArgumentCaptor.forClass(GetOutputStreamCommand.class); + verify(mockExecutor).executeCommand(captor.capture()); + assertThat(captor.getValue().key, is(KEY)); + assertThat(captor.getValue().domain, is(DOMAIN)); + assertThat(captor.getValue().storageClass, is(STORAGE_CLASS)); + } + + @Test + public void getOutputStreamCommandReturn() throws IOException { + URL path = new URL("http://localhost:80/"); + Destination destination = new Destination(path, 2, 4); + List destinations = Collections.singletonList(destination); + when(mockTracker.createOpen(KEY, DOMAIN, STORAGE_CLASS)).thenReturn(destinations); + when(mockHttpFactory.newConnection(path)).thenReturn(mockUrlConnection); + when(mockUrlConnection.getOutputStream()).thenReturn(mockOutputStream); + + // check that whatever we have delegates to the expected stream + OutputStream outputStream = file.getOutputStream(); + byte[] myBuffer = new byte[2]; + outputStream.write(myBuffer, 3, 5); + verify(mockOutputStream).write(myBuffer, 3, 5); + } + + @Test + public void getPaths() throws IOException { + List trackerPaths = Collections.singletonList(new URL("http://www.last.fm/1/2")); + when(mockTracker.getPaths(KEY, DOMAIN)).thenReturn(trackerPaths); + List paths = file.getPaths(); + assertThat(paths, is(trackerPaths)); + } + +} diff --git a/src/test/java/fm/last/moji/impl/RenameCommandTest.java b/src/test/java/fm/last/moji/impl/RenameCommandTest.java new file mode 100644 index 0000000..051ff90 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/RenameCommandTest.java @@ -0,0 +1,46 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.verify; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Tracker; + +@RunWith(MockitoJUnitRunner.class) +public class RenameCommandTest { + + @Mock + private Tracker mockTracker; + private RenameCommand command; + + @Before + public void init() { + command = new RenameCommand("key", "domain", "newKey"); + } + + @Test + public void delegatesToTracker() throws Exception { + command.executeWithTracker(mockTracker); + verify(mockTracker).rename("key", "domain", "newKey"); + } + +} diff --git a/src/test/java/fm/last/moji/impl/UpdateStorageClassCommandTest.java b/src/test/java/fm/last/moji/impl/UpdateStorageClassCommandTest.java new file mode 100644 index 0000000..367c189 --- /dev/null +++ b/src/test/java/fm/last/moji/impl/UpdateStorageClassCommandTest.java @@ -0,0 +1,46 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.impl; + +import static org.mockito.Mockito.verify; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Tracker; + +@RunWith(MockitoJUnitRunner.class) +public class UpdateStorageClassCommandTest { + + @Mock + private Tracker mockTracker; + private UpdateStorageClassCommand command; + + @Before + public void init() { + command = new UpdateStorageClassCommand("key", "domain", "newStorageClass"); + } + + @Test + public void delegatesToTracker() throws Exception { + command.executeWithTracker(mockTracker); + verify(mockTracker).updateStorageClass("key", "domain", "newStorageClass"); + } + +} diff --git a/src/test/java/fm/last/moji/integration/AbstractMojiIT.java b/src/test/java/fm/last/moji/integration/AbstractMojiIT.java new file mode 100644 index 0000000..080ef47 --- /dev/null +++ b/src/test/java/fm/last/moji/integration/AbstractMojiIT.java @@ -0,0 +1,133 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.integration; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.StringReader; +import java.io.StringWriter; +import java.util.ArrayList; +import java.util.List; +import java.util.Properties; + +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.RandomStringUtils; +import org.apache.commons.lang.math.RandomUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Ignore; + +import fm.last.moji.MojiFile; +import fm.last.moji.spring.SpringMojiBean; +import fm.last.moji.tracker.UnknownKeyException; + +@Ignore("abstract") +abstract public class AbstractMojiIT { + + private final List mojiFiles = new ArrayList(); + + SpringMojiBean moji; + String keyPrefix; + String storageClassA; + String storageClassB; + + @Before + public void setUp() throws Exception { + String env = System.getProperty("env", ""); + if (!"".equals(env)) { + env = "." + env; + } + Properties properties = new Properties(); + properties.load(getClass().getResourceAsStream("/moji.properties" + env)); + + String hosts = properties.getProperty("moji.tracker.hosts"); + String domain = properties.getProperty("moji.domain"); + + keyPrefix = properties.getProperty("test.moji.key.prefix"); + storageClassA = properties.getProperty("test.moji.class.a"); + storageClassB = properties.getProperty("test.moji.class.b"); + + moji = new SpringMojiBean(hosts, domain); + moji.setTestOnBorrow(true); + } + + @After + public void tearDown() throws Exception { + for (MojiFile file : mojiFiles) { + if (file != null) { + try { + file.delete(); + } catch (UnknownKeyException e) { + } + } + } + moji.close(); + } + + void writeDataToMogileFile(MojiFile file, String data) throws IOException { + OutputStream streamToMogile = null; + StringReader toUpload = null; + try { + toUpload = new StringReader(data); + streamToMogile = file.getOutputStream(); + IOUtils.copy(toUpload, streamToMogile); + streamToMogile.flush(); + } finally { + IOUtils.closeQuietly(streamToMogile); + IOUtils.closeQuietly(toUpload); + } + } + + String downloadDataFromMogileFile(MojiFile file) throws IOException { + StringWriter downloaded = null; + InputStream streamFromMogile = null; + try { + downloaded = new StringWriter(); + streamFromMogile = file.getInputStream(); + IOUtils.copy(streamFromMogile, downloaded); + } finally { + IOUtils.closeQuietly(downloaded); + IOUtils.closeQuietly(streamFromMogile); + } + return downloaded.toString(); + } + + MojiFile getFile(String key) { + MojiFile file = moji.getFile(key); + mojiFiles.add(file); + return file; + } + + MojiFile getFile(String key, String storageClass) { + MojiFile file = moji.getFile(key, storageClass); + mojiFiles.add(file); + return file; + } + + String newData() { + return RandomStringUtils.randomAscii(RandomUtils.nextInt(4096) + 512); + } + + String newKey(String suffix) { + return keyPrefix + suffix; + } + + String newKey() { + return newKey(RandomStringUtils.randomAlphanumeric(16)); + } + +} diff --git a/src/test/java/fm/last/moji/integration/MojiFileIT.java b/src/test/java/fm/last/moji/integration/MojiFileIT.java new file mode 100644 index 0000000..2789deb --- /dev/null +++ b/src/test/java/fm/last/moji/integration/MojiFileIT.java @@ -0,0 +1,203 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.integration; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.IOException; +import java.net.URL; +import java.util.List; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.lang.RandomStringUtils; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import fm.last.moji.MojiFile; +import fm.last.moji.tracker.KeyExistsAlreadyException; +import fm.last.moji.tracker.UnknownKeyException; +import fm.last.moji.tracker.UnknownStorageClassException; + +public class MojiFileIT extends AbstractMojiIT { + + @Rule + public TemporaryFolder testFolder = new TemporaryFolder(); + + @Test + public void writeNewThenReadBack() throws Exception { + MojiFile newFile = getFile(newKey()); + assertFalse(newFile.exists()); + + String data = newData(); + + writeDataToMogileFile(newFile, data); + assertTrue(newFile.exists()); + + assertEquals(data, downloadDataFromMogileFile(newFile)); + } + + @Test + public void overwriteThenReadBack() throws Exception { + MojiFile existingFile = getFile(newKey("overwriteThenReadBack")); + String overwrite = newData(); + + writeDataToMogileFile(existingFile, overwrite); + assertEquals(overwrite, downloadDataFromMogileFile(existingFile)); + } + + @Test + public void writeNewWithStorageClassThenReadBack() throws Exception { + MojiFile newFile = getFile(newKey(), storageClassA); + assertFalse(newFile.exists()); + + String data = newData(); + + writeDataToMogileFile(newFile, data); + assertTrue(newFile.exists()); + // We cannot know the storage class currently + // assertEquals(newFile.getStorageClass(), storageClassA); + + assertEquals(data, downloadDataFromMogileFile(newFile)); + } + + @Test(expected = UnknownStorageClassException.class) + public void writeWithstorageClassUnknown() throws IOException { + MojiFile fileInUnknownClass = getFile(newKey(), "madeup" + RandomStringUtils.randomAlphanumeric(8)); + assertFalse(fileInUnknownClass.exists()); + + String data = newData(); + + writeDataToMogileFile(fileInUnknownClass, data); + } + + @Test + public void fileSize() throws IOException { + MojiFile fileWithSize = getFile(newKey("fileOfKnownSize")); + assertEquals(3832, fileWithSize.length()); + } + + @Test + public void exists() throws IOException { + MojiFile existentFile = getFile(newKey("exists")); + assertTrue(existentFile.exists()); + } + + @Test + public void notExists() throws IOException { + MojiFile existentFile = getFile(newKey()); + assertFalse(existentFile.exists()); + } + + @Test + public void notExistsAfterDelete() throws IOException { + MojiFile existentFile = getFile(newKey("notExistsAfterDelete")); + assertTrue(existentFile.exists()); + existentFile.delete(); + assertFalse(existentFile.exists()); + } + + @Test + public void rename() throws IOException { + String originalKey = newKey("rename"); + MojiFile fileToRename = getFile(originalKey); + + String newKey = newKey(); + fileToRename.rename(newKey); + assertEquals(newKey, fileToRename.getKey()); + + MojiFile renamed = getFile(newKey); + assertTrue(renamed.exists()); + + MojiFile oldName = getFile(originalKey); + assertFalse(oldName.exists()); + } + + @Test(expected = UnknownKeyException.class) + public void renameUnknownKey() throws IOException { + MojiFile nonExistentFile = getFile(newKey()); + nonExistentFile.rename(newKey()); + } + + @Test(expected = KeyExistsAlreadyException.class) + public void renameExistingKey() throws IOException { + String alreadyHereKey = newKey("renameExistingKey1"); + String toRenameKey = newKey("renameExistingKey2"); + + MojiFile newFile = getFile(toRenameKey); + newFile.rename(alreadyHereKey); + } + + @Test + public void updateStorageClass() throws IOException { + String key = newKey("updateStorageClass"); + MojiFile fileToUpdate = getFile(key, storageClassA); + // We cannot know the storage class currently + // assertEquals(fileToUpdate.getStorageClass(), storageClassA); + + fileToUpdate.modifyStorageClass(storageClassB); + // We cannot know the storage class currently + // assertEquals(fileToUpdate.getStorageClass(), storageClassB); + + MojiFile exists = getFile(key, storageClassB); + assertTrue(exists.exists()); + } + + @Test(expected = UnknownStorageClassException.class) + public void updateStorageClassToUnknown() throws IOException { + MojiFile fileToUpdate = getFile(newKey("updateStorageClassToUnknown")); + fileToUpdate.modifyStorageClass("madeup" + RandomStringUtils.randomAlphanumeric(8)); + } + + @Test(expected = UnknownKeyException.class) + public void deleteNonExistent() throws IOException { + MojiFile fileToDelete = getFile(newKey()); + fileToDelete.delete(); + } + + @Test + public void copyToFile() throws IOException { + MojiFile copyFile = getFile(newKey("mogileFileCopyToFile")); + + File file = testFolder.newFile(newKey() + ".dat"); + copyFile.copyToFile(file); + + byte[] actualData = FileUtils.readFileToByteArray(file); + + File original = new File("src/test/data/mogileFileCopyToFile.dat"); + byte[] expectedData = FileUtils.readFileToByteArray(original); + assertArrayEquals(expectedData, actualData); + } + + @Test(expected = UnknownKeyException.class) + public void copyToFileUnknownKey() throws IOException { + MojiFile copyFile = getFile(newKey()); + File file = testFolder.newFile(newKey() + ".dat"); + copyFile.copyToFile(file); + } + + @Test + public void getPaths() throws Exception { + MojiFile file = getFile(newKey("getPaths")); + List paths = file.getPaths(); + assertFalse(paths.isEmpty()); + } + +} diff --git a/src/test/java/fm/last/moji/integration/MojiIT.java b/src/test/java/fm/last/moji/integration/MojiIT.java new file mode 100644 index 0000000..20f1a1e --- /dev/null +++ b/src/test/java/fm/last/moji/integration/MojiIT.java @@ -0,0 +1,79 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.integration; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThat; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.IOException; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import org.apache.commons.io.FileUtils; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import fm.last.moji.MojiFile; + +public class MojiIT extends AbstractMojiIT { + + @Rule + public TemporaryFolder testFolder = new TemporaryFolder(); + + @Test + public void copyToMogile() throws IOException { + File tempFile = testFolder.newFile(newKey() + ".txt"); + String data = newData(); + + FileUtils.write(tempFile, data); + MojiFile destination = getFile(newKey()); + + moji.copyToMogile(tempFile, destination); + + assertEquals(data, downloadDataFromMogileFile(destination)); + } + + @Test + public void list() throws IOException { + List list = moji.list(keyPrefix + "list"); + assertThat(list.size(), is(3)); + Set keys = new HashSet(); + for (MojiFile mojiFile : list) { + keys.add(mojiFile.getKey()); + } + assertTrue(keys.contains(keyPrefix + "list1")); + assertTrue(keys.contains(keyPrefix + "list2")); + assertTrue(keys.contains(keyPrefix + "list3")); + } + + @Test + public void listWithLimit() throws IOException { + List list = moji.list(keyPrefix + "list", 1); + assertThat(list.size(), is(1)); + } + + @Test + public void listNoMatches() throws IOException { + List list = moji.list("XXX" + keyPrefix); + assertTrue(list.isEmpty()); + } + +} diff --git a/src/test/java/fm/last/moji/local/DefaultFileNamingStrategyTest.java b/src/test/java/fm/last/moji/local/DefaultFileNamingStrategyTest.java new file mode 100644 index 0000000..24cbd5d --- /dev/null +++ b/src/test/java/fm/last/moji/local/DefaultFileNamingStrategyTest.java @@ -0,0 +1,70 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.local; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertThat; +import static org.junit.Assert.assertTrue; + +import java.io.File; +import java.io.FilenameFilter; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class DefaultFileNamingStrategyTest { + + @Rule + public TemporaryFolder testFolder = new TemporaryFolder(); + + private File folder; + private DefaultFileNamingStrategy namingStrategy; + + @Before + public void setUp() { + folder = testFolder.newFolder("local-moji"); + namingStrategy = new DefaultFileNamingStrategy(folder); + } + + @Test + public void keyForFile() { + String key = namingStrategy.keyForFileName("lastfm-8473848737.dat"); + assertThat(key, is("8473848737")); + } + + @Test + public void domainForFile() { + String domain = namingStrategy.domainForFileName("lastfm-8473848737.dat"); + assertThat(domain, is("lastfm")); + } + + @Test + public void fileNameFilter() { + File anotherFolder = testFolder.newFolder("some-other-folder"); + FilenameFilter filter = namingStrategy.filterForPrefix("lastfm", "100"); + assertTrue(filter.accept(folder, "lastfm-1003848737.dat")); + assertFalse(filter.accept(folder, "lastfm-1013848737.dat")); + assertFalse(filter.accept(folder, "another-1003848737.dat")); + assertFalse(filter.accept(folder, ".ssh")); + assertFalse(filter.accept(folder, "..")); + assertFalse(filter.accept(folder, ".")); + assertFalse(filter.accept(anotherFolder, "lastfm-1003848737.dat")); + } + +} diff --git a/src/test/java/fm/last/moji/spring/SpringMojiBeanInstantiationTest.java b/src/test/java/fm/last/moji/spring/SpringMojiBeanInstantiationTest.java new file mode 100644 index 0000000..18b5f9d --- /dev/null +++ b/src/test/java/fm/last/moji/spring/SpringMojiBeanInstantiationTest.java @@ -0,0 +1,44 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.spring; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; + +import org.junit.Test; + +import fm.last.moji.FakeMogileFsServer; +import fm.last.moji.MojiFile; + +public class SpringMojiBeanInstantiationTest { + + @Test(timeout = 2000) + public void delete() throws Exception { + FakeMogileFsServer server = null; + try { + FakeMogileFsServer.Builder builder = new FakeMogileFsServer.Builder(); + builder.whenRequestContains("delete ", "key=myKey", "domain=myDomain").thenRespond("OK "); + server = builder.build(); + SpringMojiBean bean = new SpringMojiBean(server.getAddressAsString(), "myDomain"); + MojiFile file = bean.getFile("myKey"); + file.delete(); + assertThat(bean.getNumIdle(), is(1)); + } finally { + server.close(); + } + } + +} diff --git a/src/test/java/fm/last/moji/spring/SpringMojiBeanTest.java b/src/test/java/fm/last/moji/spring/SpringMojiBeanTest.java new file mode 100644 index 0000000..250e3b6 --- /dev/null +++ b/src/test/java/fm/last/moji/spring/SpringMojiBeanTest.java @@ -0,0 +1,67 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.spring; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; + +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.util.Set; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +public class SpringMojiBeanTest { + + private SpringMojiBean bean; + + @Before + public void init() { + bean = new SpringMojiBean("localhost:80", "domain"); + } + + @After + public void close() throws Exception { + bean.close(); + } + + @Test + public void defaultProxy() { + assertEquals(Proxy.NO_PROXY, bean.getProxy()); + } + + @Test + public void address() { + Set addresses = bean.getAddresses(); + assertEquals(1, addresses.size()); + InetSocketAddress address = addresses.iterator().next(); + assertEquals("localhost", address.getHostName()); + assertEquals(address.getPort(), 80); + } + + @Test + public void getFile() { + assertNotNull(bean.getFile("123")); + } + + @Test + public void getFileWithStorageClass() { + assertNotNull(bean.getFile("123", "class")); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/CreateOpenOperationTest.java b/src/test/java/fm/last/moji/tracker/impl/CreateOpenOperationTest.java new file mode 100644 index 0000000..57f311c --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/CreateOpenOperationTest.java @@ -0,0 +1,182 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; +import static org.mockito.Mockito.when; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.TrackerException; + +@RunWith(MockitoJUnitRunner.class) +public class CreateOpenOperationTest { + + private static final String DOMAIN = "domain"; + private static final String STORAGE_CLASS = "storageClass"; + private static final String KEY = "key"; + private static final String FID = "5"; + private static final String DEV_COUNT = "2"; + private static final String PATH_1 = "http://www.last.fm/1/"; + private static final String DEV_ID_1 = "1"; + private static final String PATH_2 = "http://www.last.fm/2/"; + private static final String DEV_ID_2 = "2"; + + @Mock + private RequestHandler mockRequestHandler; + @Mock + private Response mockResponse; + @Mock + private Response mockEmptyResponse; + @Mock + private Response mockUnknownKeyResponse; + @Mock + private Response mockFailResponse; + + private ArgumentCaptor captorRequest; + private CreateOpenOperation operation; + + @Before + public void setUp() throws TrackerException { + captorRequest = ArgumentCaptor.forClass(Request.class); + when(mockResponse.getStatus()).thenReturn(ResponseStatus.OK); + when(mockResponse.getValue("fid")).thenReturn(FID); + when(mockResponse.getValue("dev_count")).thenReturn(DEV_COUNT); + when(mockResponse.getValue("path_1")).thenReturn(PATH_1); + when(mockResponse.getValue("devid_1")).thenReturn(DEV_ID_1); + when(mockResponse.getValue("path_2")).thenReturn(PATH_2); + when(mockResponse.getValue("devid_2")).thenReturn(DEV_ID_2); + + when(mockEmptyResponse.getStatus()).thenReturn(ResponseStatus.OK); + when(mockEmptyResponse.getValue("fid")).thenReturn(FID); + when(mockEmptyResponse.getValue("dev_count")).thenReturn("0"); + + when(mockUnknownKeyResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockUnknownKeyResponse.getMessage()).thenReturn("unknown_key unknown key"); + + when(mockFailResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockFailResponse.getMessage()).thenReturn("unexpected error"); + } + + @Test + public void requestNormal() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, STORAGE_CLASS, true); + operation.execute(); + + Request request = captorRequest.getValue(); + assertThat(request.getCommand(), is("create_open")); + assertThat(request.getArguments().size(), is(4)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("class"), is(STORAGE_CLASS)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("multi_dest"), is("1")); + } + + @Test + public void requestNormalNoClass() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, null, true); + operation.execute(); + + Request request = captorRequest.getValue(); + assertThat(request.getCommand(), is("create_open")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("multi_dest"), is("1")); + } + + @Test + public void requestNormalNoMulti() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, "", false); + operation.execute(); + + Request request = captorRequest.getValue(); + assertThat(request.getCommand(), is("create_open")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("multi_dest"), is("0")); + } + + @Test + public void responseNormal() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, STORAGE_CLASS, true); + operation.execute(); + + List destinations = operation.getDestinations(); + assertThat(destinations.size(), is(2)); + + Destination destination1 = destinations.get(0); + assertThat(destination1.getDevId(), is(1)); + assertThat(destination1.getFid(), is(5)); + assertThat(destination1.getPath(), is(new URL(PATH_1))); + + Destination destination2 = destinations.get(1); + assertThat(destination2.getDevId(), is(2)); + assertThat(destination2.getFid(), is(5)); + assertThat(destination2.getPath(), is(new URL(PATH_2))); + } + + @Test + public void zeroDestinations() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockEmptyResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, STORAGE_CLASS, true); + operation.execute(); + + List destinations = operation.getDestinations(); + assertThat(destinations.size(), is(0)); + } + + @Test + public void unknownKey() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockUnknownKeyResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, STORAGE_CLASS, true); + operation.execute(); + + List destinations = operation.getDestinations(); + assertThat(destinations.size(), is(0)); + } + + @Test(expected = TrackerException.class) + public void unexpectedError() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockFailResponse); + + operation = new CreateOpenOperation(mockRequestHandler, DOMAIN, KEY, STORAGE_CLASS, true); + operation.execute(); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/GetPathsOperationTest.java b/src/test/java/fm/last/moji/tracker/impl/GetPathsOperationTest.java new file mode 100644 index 0000000..8db247f --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/GetPathsOperationTest.java @@ -0,0 +1,147 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; +import static org.mockito.Mockito.when; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.UnknownKeyException; + +@RunWith(MockitoJUnitRunner.class) +public class GetPathsOperationTest { + + private static final String DOMAIN = "domain"; + private static final String KEY = "key"; + private static final String PATH_COUNT = "2"; + private static final String PATH_1 = "http://www.last.fm/1/"; + private static final String PATH_2 = "http://www.last.fm/2/"; + + @Mock + private RequestHandler mockRequestHandler; + @Mock + private Response mockResponse; + @Mock + private Response mockEmptyResponse; + @Mock + private Response mockUnknownKeyResponse; + @Mock + private Response mockFailResponse; + + private ArgumentCaptor captorRequest; + private GetPathsOperation operation; + + @Before + public void setUp() throws TrackerException { + captorRequest = ArgumentCaptor.forClass(Request.class); + when(mockResponse.getStatus()).thenReturn(ResponseStatus.OK); + when(mockResponse.getValue("paths")).thenReturn(PATH_COUNT); + when(mockResponse.getValue("path1")).thenReturn(PATH_1); + when(mockResponse.getValue("path2")).thenReturn(PATH_2); + + when(mockEmptyResponse.getStatus()).thenReturn(ResponseStatus.OK); + when(mockEmptyResponse.getValue("paths")).thenReturn("0"); + + when(mockUnknownKeyResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockUnknownKeyResponse.getMessage()).thenReturn("unknown_key unknown key"); + + when(mockFailResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockFailResponse.getMessage()).thenReturn("unexpected error"); + } + + @Test + public void requestNormal() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, false); + operation.execute(); + + Request request = captorRequest.getValue(); + assertThat(request.getCommand(), is("get_paths")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("noverify"), is("1")); + } + + @Test + public void requestNormalVerify() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, true); + operation.execute(); + + Request request = captorRequest.getValue(); + assertThat(request.getCommand(), is("get_paths")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("noverify"), is("0")); + } + + @Test + public void responseNormal() throws TrackerException, MalformedURLException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, true); + operation.execute(); + + List paths = operation.getPaths(); + assertThat(paths.size(), is(2)); + assertThat(paths.get(0), is(new URL(PATH_1))); + assertThat(paths.get(1), is(new URL(PATH_2))); + } + + @Test + public void zeroPaths() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockEmptyResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, true); + operation.execute(); + + List paths = operation.getPaths(); + assertThat(paths.size(), is(0)); + } + + @Test(expected = UnknownKeyException.class) + public void unknownKey() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockUnknownKeyResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, true); + operation.execute(); + } + + @Test(expected = TrackerException.class) + public void unexpectedError() throws TrackerException { + when(mockRequestHandler.performRequest(captorRequest.capture())).thenReturn(mockFailResponse); + + operation = new GetPathsOperation(mockRequestHandler, DOMAIN, KEY, true); + operation.execute(); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.java b/src/test/java/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.java new file mode 100644 index 0000000..8568076 --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.java @@ -0,0 +1,79 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.junit.Assert.assertEquals; + +import java.net.InetSocketAddress; + +import org.junit.Test; + +public class InetSocketAddressFactoryTest { + + @Test + public void numeric1() { + InetSocketAddress address = InetSocketAddressFactory.newAddress("0.0.0.0:80"); + assertEquals("0.0.0.0", address.getHostName()); + assertEquals(80, address.getPort()); + } + + @Test + public void numeric2() { + InetSocketAddress address = InetSocketAddressFactory.newAddress("255.255.255.255:65535"); + assertEquals("255.255.255.255", address.getHostName()); + assertEquals(65535, address.getPort()); + } + + @Test(expected = IllegalArgumentException.class) + public void numericBadPort1() { + InetSocketAddressFactory.newAddress("255.255.255.255:65536"); + } + + @Test(expected = IllegalArgumentException.class) + public void numericBadAddress1() { + InetSocketAddressFactory.newAddress("255.255.255.256:65535"); + } + + @Test(expected = IllegalArgumentException.class) + public void numericBadPort2() { + InetSocketAddressFactory.newAddress("255.255.255.255:-80"); + } + + @Test(expected = IllegalArgumentException.class) + public void numericBadAddress2() { + InetSocketAddressFactory.newAddress("255.-2.255.255:80"); + } + + @Test + public void alpha1() { + InetSocketAddress address = InetSocketAddressFactory.newAddress("www.google.com:80"); + assertEquals("www.google.com", address.getHostName()); + assertEquals(80, address.getPort()); + } + + @Test + public void alpha2() { + InetSocketAddress address = InetSocketAddressFactory.newAddress("localhost:80"); + assertEquals("localhost", address.getHostName()); + assertEquals(80, address.getPort()); + } + + @Test(expected = IllegalArgumentException.class) + public void badAlpha1() { + InetSocketAddressFactory.newAddress("local:host:80"); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/RequestHandlerTest.java b/src/test/java/fm/last/moji/tracker/impl/RequestHandlerTest.java new file mode 100644 index 0000000..0361bfb --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/RequestHandlerTest.java @@ -0,0 +1,156 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.mockito.Matchers.anyChar; +import static org.mockito.Matchers.anyInt; +import static org.mockito.Matchers.argThat; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.StringReader; +import java.io.StringWriter; +import java.io.Writer; + +import org.hamcrest.BaseMatcher; +import org.hamcrest.Description; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.TrackerException; + +@RunWith(MockitoJUnitRunner.class) +public class RequestHandlerTest { + + @Mock + private Writer mockWriter; + @Mock + private BufferedReader mockReader; + private RequestHandler handler; + + @Test + public void write() throws TrackerException { + BufferedReader reader = new BufferedReader(new StringReader("OK r1=x&r2=y")); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + handler.performRequest(request); + + assertEquals("mock arg1=one&arg2=2\r\n", writer.toString()); + } + + @Test + public void okRead() throws TrackerException { + BufferedReader reader = new BufferedReader(new StringReader("OK r1=x&r2=y")); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + Response response = handler.performRequest(request); + + assertEquals(ResponseStatus.OK, response.getStatus()); + assertEquals("x", response.getValue("r1")); + assertEquals("y", response.getValue("r2")); + assertNull(response.getMessage()); + } + + @Test + public void errorRead() throws TrackerException { + BufferedReader reader = new BufferedReader(new StringReader("ERR problem")); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + Response response = handler.performRequest(request); + + assertEquals(ResponseStatus.ERROR, response.getStatus()); + assertEquals("problem", response.getMessage()); + } + + @Test(expected = TrackerException.class) + public void badResponse() throws TrackerException { + BufferedReader reader = new BufferedReader(new StringReader("ERR")); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + handler.performRequest(request); + } + + @Test(expected = TrackerException.class) + public void badResponse2() throws TrackerException { + BufferedReader reader = new BufferedReader(new StringReader("XXX problem")); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + handler.performRequest(request); + } + + @Test(expected = TrackerException.class) + public void ioExceptionRead() throws IOException { + when(mockReader.readLine()).thenThrow(new IOException()); + StringWriter writer = new StringWriter(); + handler = new RequestHandler(writer, mockReader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + handler.performRequest(request); + } + + @Test(expected = TrackerException.class) + public void ioExceptionWrite() throws IOException { + doThrow(new IOException()).when(mockWriter).write(anyInt()); + doThrow(new IOException()).when(mockWriter).write(anyChar()); + doThrow(new IOException()).when(mockWriter).write(argThat(new TrueMatcher())); + doThrow(new IOException()).when(mockWriter).write(argThat(new TrueMatcher())); + doThrow(new IOException()).when(mockWriter).write(argThat(new TrueMatcher()), anyInt(), anyInt()); + doThrow(new IOException()).when(mockWriter).write(argThat(new TrueMatcher()), anyInt(), anyInt()); + BufferedReader reader = new BufferedReader(new StringReader("ERR problem")); + handler = new RequestHandler(mockWriter, reader); + + Request request = new Request.Builder(2).command("mock").arg("arg1", "one").arg("arg2", 2).build(); + handler.performRequest(request); + } + + @Test + public void close() throws IOException { + handler = new RequestHandler(mockWriter, mockReader); + handler.close(); + verify(mockReader).close(); + verify(mockWriter).close(); + } + + private class TrueMatcher extends BaseMatcher { + + @Override + public boolean matches(Object arg0) { + return true; + } + + @Override + public void describeTo(Description arg0) { + } + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/RequestTest.java b/src/test/java/fm/last/moji/tracker/impl/RequestTest.java new file mode 100644 index 0000000..46b5153 --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/RequestTest.java @@ -0,0 +1,42 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.junit.Assert.assertTrue; + +import java.io.StringWriter; +import java.net.URL; + +import org.junit.Test; + +public class RequestTest { + + @Test + public void example() throws Exception { + Request request = new Request.Builder(5).command("mycommand").arg("bool", true).arg("integer", 2).arg("long", 12L) + .arg("string", "/=&URL").arg("url", new URL("http://localhost:80/x.do?what=12&do")).build(); + StringWriter writer = new StringWriter(); + request.writeTo(writer); + String wire = writer.toString(); + assertTrue(wire.startsWith("mycommand ")); + assertTrue(wire.contains("bool=1")); + assertTrue(wire.contains("long=12")); + assertTrue(wire.contains("integer=2")); + assertTrue(wire.contains("url=http%3A%2F%2Flocalhost%3A80%2Fx.do%3Fwhat%3D12%26do")); + assertTrue(wire.contains("string=%2F%3D%26URL")); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/ResponseTest.java b/src/test/java/fm/last/moji/tracker/impl/ResponseTest.java new file mode 100644 index 0000000..fadd8a8 --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/ResponseTest.java @@ -0,0 +1,57 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.CoreMatchers.nullValue; +import static org.junit.Assert.assertThat; + +import org.junit.Test; + +public class ResponseTest { + + @Test + public void example() { + Response response = new Response(ResponseStatus.OK, + "path2=http://127.0.0.61:7500/dev469/0/173/153/0173153702.fid&path1=" + + "http://127.0.0.62:7500/dev490/0/173/153/0173153702.fid&paths=2"); + assertThat(response.getStatus(), is(ResponseStatus.OK)); + assertThat(response.getValue("paths"), is("2")); + assertThat(response.getValue("path1"), is("http://127.0.0.62:7500/dev490/0/173/153/0173153702.fid")); + assertThat(response.getValue("path2"), is("http://127.0.0.61:7500/dev469/0/173/153/0173153702.fid")); + assertThat(response.getMessage(), is(nullValue())); + } + + @Test + public void badPair() { + Response response = new Response(ResponseStatus.OK, + "path2=http://127.0.0.61:7500/dev469/0/173/153/0173153702.fid&path1" + + "http://127.0.0.62:7500/dev490/0/173/153/0173153702.fid&paths=2"); + assertThat(response.getStatus(), is(ResponseStatus.OK)); + assertThat(response.getValue("paths"), is("2")); + assertThat(response.getValue("path1"), is(nullValue())); + assertThat(response.getValue("path2"), is("http://127.0.0.61:7500/dev469/0/173/153/0173153702.fid")); + assertThat(response.getMessage(), is(nullValue())); + } + + @Test + public void error() { + Response response = new Response(ResponseStatus.ERROR, "message"); + assertThat(response.getStatus(), is(ResponseStatus.ERROR)); + assertThat(response.getMessage(), is("message")); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.java b/src/test/java/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.java new file mode 100644 index 0000000..61386ed --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.java @@ -0,0 +1,72 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.net.Proxy; +import java.net.ServerSocket; +import java.util.Set; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; + +public class SingleHostTrackerFactoryTest { + + private SingleHostTrackerFactory factory; + private ServerSocket serverSocket; + private InetSocketAddress address; + + @Before + public void init() throws IOException { + serverSocket = new ServerSocket(0); + address = new InetSocketAddress(serverSocket.getInetAddress(), serverSocket.getLocalPort()); + factory = new SingleHostTrackerFactory(address, Proxy.NO_PROXY); + } + + @After + public void tearDown() throws IOException { + serverSocket.close(); + } + + @Test + public void proxy() { + assertEquals(Proxy.NO_PROXY, factory.getProxy()); + } + + @Test + public void getTracker() throws TrackerException { + Tracker tracker = factory.getTracker(); + assertNotNull(tracker); + } + + @Test + public void getAddresses() { + Set addresses = factory.getAddresses(); + assertEquals(1, addresses.size()); + InetSocketAddress actualAddress = addresses.iterator().next(); + assertEquals(address.getHostName(), actualAddress.getHostName()); + assertEquals(address.getPort(), actualAddress.getPort()); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/impl/TrackerImplTest.java b/src/test/java/fm/last/moji/tracker/impl/TrackerImplTest.java new file mode 100644 index 0000000..d08e377 --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/impl/TrackerImplTest.java @@ -0,0 +1,218 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.impl; + +import static org.hamcrest.CoreMatchers.is; +import static org.junit.Assert.assertThat; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.net.Socket; +import java.net.URL; +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.KeyExistsAlreadyException; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.UnknownKeyException; + +@RunWith(MockitoJUnitRunner.class) +public class TrackerImplTest { + + private static final String STORAGE_CLASS = "storageClass"; + private static final String KEY = "key"; + private static final String DOMAIN = "domain"; + private static final long SIZE = 0; + private static final String NEW_KEY = "newKey"; + + @Mock + private Socket mockSocket; + @Mock + private RequestHandler mockRequestHandler; + @Mock + private Response mockResponse; + + private TrackerImpl tracker; + private final ArgumentCaptor requestCaptor = ArgumentCaptor.forClass(Request.class); + + @Before + public void setUp() throws IOException { + tracker = new TrackerImpl(mockSocket, mockRequestHandler); + when(mockResponse.getStatus()).thenReturn(ResponseStatus.OK); + when(mockRequestHandler.performRequest(requestCaptor.capture())).thenReturn(mockResponse); + } + + @Test + public void getPaths() throws Exception { + // See: GetPathsOperationTest + when(mockResponse.getValue("paths")).thenReturn("0"); + List paths = tracker.getPaths(KEY, DOMAIN); + assertThat(paths.size(), is(0)); + } + + @Test + public void createOpen() throws Exception { + // See: CreateOpenOperationTest + when(mockResponse.getValue("dev_count")).thenReturn("0"); + when(mockResponse.getValue("fid")).thenReturn("0"); + List destinations = tracker.createOpen(KEY, DOMAIN, STORAGE_CLASS); + assertThat(destinations.size(), is(0)); + } + + @Test + public void deleteRequest() throws Exception { + tracker.delete(KEY, DOMAIN); + Request request = requestCaptor.getValue(); + assertThat(request.getCommand(), is("delete")); + assertThat(request.getArguments().size(), is(2)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + } + + @Test(expected = UnknownKeyException.class) + public void deleteUnknownKey() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("unknown_key"); + tracker.delete(KEY, DOMAIN); + } + + @Test(expected = TrackerException.class) + public void deleteFails() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("something else"); + try { + tracker.delete(KEY, DOMAIN); + } catch (UnknownKeyException ignored) { + } + } + + @Test + public void updateStorageClassRequest() throws Exception { + tracker.updateStorageClass(KEY, DOMAIN, STORAGE_CLASS); + Request request = requestCaptor.getValue(); + assertThat(request.getCommand(), is("updateclass")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("class"), is(STORAGE_CLASS)); + } + + @Test(expected = UnknownKeyException.class) + public void updateStorageClassUnknownKey() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("unknown_key"); + tracker.updateStorageClass(KEY, DOMAIN, STORAGE_CLASS); + } + + @Test(expected = TrackerException.class) + public void updateStorageClassFails() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("something else"); + try { + tracker.updateStorageClass(KEY, DOMAIN, STORAGE_CLASS); + } catch (UnknownKeyException ignored) { + } + } + + @Test + public void renameRequest() throws Exception { + tracker.rename(KEY, DOMAIN, NEW_KEY); + Request request = requestCaptor.getValue(); + assertThat(request.getCommand(), is("rename")); + assertThat(request.getArguments().size(), is(3)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("from_key"), is(KEY)); + assertThat(request.getArguments().get("to_key"), is(NEW_KEY)); + } + + @Test(expected = UnknownKeyException.class) + public void renameUnknownKey() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("unknown_key"); + tracker.rename(KEY, DOMAIN, NEW_KEY); + } + + @Test(expected = KeyExistsAlreadyException.class) + public void renameKeyExists() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("key_exists"); + tracker.rename(KEY, DOMAIN, NEW_KEY); + } + + @Test(expected = TrackerException.class) + public void renameFails() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("something else"); + try { + tracker.rename(KEY, DOMAIN, NEW_KEY); + } catch (UnknownKeyException ignored) { + } catch (KeyExistsAlreadyException ignored) { + } + } + + @Test + public void noopRequest() throws Exception { + tracker.noop(); + Request request = requestCaptor.getValue(); + assertThat(request.getCommand(), is("noop")); + assertThat(request.getArguments().size(), is(0)); + } + + @Test(expected = TrackerException.class) + public void noopFails() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("unknown_key"); + tracker.noop(); + } + + @Test + public void createCloseRequest() throws Exception { + Destination destination = new Destination(new URL("http://www.last.fm/1/"), 23, 32); + tracker.createClose(KEY, DOMAIN, destination, SIZE); + Request request = requestCaptor.getValue(); + assertThat(request.getCommand(), is("create_close")); + assertThat(request.getArguments().size(), is(6)); + assertThat(request.getArguments().get("domain"), is(DOMAIN)); + assertThat(request.getArguments().get("key"), is(KEY)); + assertThat(request.getArguments().get("size"), is(Long.toString(SIZE))); + assertThat(request.getArguments().get("devid"), is("23")); + assertThat(request.getArguments().get("path"), is("http://www.last.fm/1/")); + assertThat(request.getArguments().get("fid"), is("32")); + } + + @Test(expected = TrackerException.class) + public void createCloseFails() throws Exception { + when(mockResponse.getStatus()).thenReturn(ResponseStatus.ERROR); + when(mockResponse.getMessage()).thenReturn("unknown_key"); + tracker.noop(); + } + + @Test + public void close() throws IOException { + tracker.close(); + verify(mockSocket).close(); + verify(mockRequestHandler).close(); + } + +} diff --git a/src/test/java/fm/last/moji/tracker/pool/BorrowedTrackerTest.java b/src/test/java/fm/last/moji/tracker/pool/BorrowedTrackerTest.java new file mode 100644 index 0000000..327daa6 --- /dev/null +++ b/src/test/java/fm/last/moji/tracker/pool/BorrowedTrackerTest.java @@ -0,0 +1,126 @@ +/* + * Copyright 2009 Last.fm + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package fm.last.moji.tracker.pool; + +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyZeroInteractions; + +import org.apache.commons.pool.KeyedObjectPool; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.runners.MockitoJUnitRunner; + +import fm.last.moji.tracker.Destination; +import fm.last.moji.tracker.Tracker; +import fm.last.moji.tracker.TrackerException; +import fm.last.moji.tracker.impl.CommunicationException; + +@RunWith(MockitoJUnitRunner.class) +public class BorrowedTrackerTest { + + private static final String KEY1 = "key1"; + private static final String DOMAIN = "domain"; + private static final String KEY2 = "key2"; + private static final String STORAGE_CLASS = "class"; + private static final int SIZE = 1; + @Mock + private Tracker mockTracker; + @Mock + private KeyedObjectPool mockPool; + @Mock + private Destination mockDestination; + @Mock + private ManagedTrackerHost mockHost; + + private BorrowedTracker borrowedTracker; + + @Before + public void init() { + borrowedTracker = new BorrowedTracker(mockHost, mockTracker, mockPool); + } + + @Test + public void closeReturnsToPool() throws Exception { + borrowedTracker.close(); + verify(mockPool).returnObject(mockHost, borrowedTracker); + verifyZeroInteractions(mockTracker); + } + + @Test + public void closeWithErrorInvalidates() throws Exception { + doThrow(new CommunicationException()).when(mockTracker).noop(); + try { + borrowedTracker.noop(); + } catch (CommunicationException e) { + } + borrowedTracker.close(); + verify(mockPool).invalidateObject(mockHost, borrowedTracker); + } + + @Test + public void reallyCloseDelegates() throws Exception { + borrowedTracker.reallyClose(); + verify(mockTracker).close(); + verifyZeroInteractions(mockPool); + } + + @Test + public void getPathsDelegates() throws TrackerException { + borrowedTracker.getPaths(KEY1, DOMAIN); + verify(mockTracker).getPaths(KEY1, DOMAIN); + } + + @Test + public void createOpenDelegates() throws TrackerException { + borrowedTracker.createOpen(KEY1, DOMAIN, STORAGE_CLASS); + verify(mockTracker).createOpen(KEY1, DOMAIN, STORAGE_CLASS); + } + + @Test + public void createCloseDelegates() throws TrackerException { + borrowedTracker.createClose(KEY1, DOMAIN, mockDestination, SIZE); + verify(mockTracker).createClose(KEY1, DOMAIN, mockDestination, SIZE); + } + + @Test + public void deleteDelegates() throws TrackerException { + borrowedTracker.delete(KEY1, DOMAIN); + verify(mockTracker).delete(KEY1, DOMAIN); + } + + @Test + public void renameDelegates() throws TrackerException { + borrowedTracker.rename(KEY1, DOMAIN, KEY2); + verify(mockTracker).rename(KEY1, DOMAIN, KEY2); + + } + + @Test + public void updateStorageClassDelegates() throws TrackerException { + borrowedTracker.updateStorageClass(KEY1, DOMAIN, STORAGE_CLASS); + verify(mockTracker).updateStorageClass(KEY1, DOMAIN, STORAGE_CLASS); + } + + @Test + public void noopDelegates() throws TrackerException { + borrowedTracker.noop(); + verify(mockTracker).noop(); + } + +} diff --git a/src/test/resources/checkstyle.xml b/src/test/resources/checkstyle.xml new file mode 100644 index 0000000..5a1c437 --- /dev/null +++ b/src/test/resources/checkstyle.xml @@ -0,0 +1,142 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/findbugsExclude.xml b/src/test/resources/findbugsExclude.xml new file mode 100644 index 0000000..17344f8 --- /dev/null +++ b/src/test/resources/findbugsExclude.xml @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/src/test/resources/log4j.properties b/src/test/resources/log4j.properties new file mode 100644 index 0000000..d7f2348 --- /dev/null +++ b/src/test/resources/log4j.properties @@ -0,0 +1,8 @@ +log4j.rootLogger=INFO, stdout + +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.Target=System.out +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %5p %c{2}:%L - %m%n + +log4j.logger.fm.last = DEBUG \ No newline at end of file diff --git a/src/test/resources/moji.properties b/src/test/resources/moji.properties new file mode 100644 index 0000000..daa6075 --- /dev/null +++ b/src/test/resources/moji.properties @@ -0,0 +1,7 @@ +# My local properties +moji.tracker.hosts=localhost:7001 +moji.domain=testdomain + +test.moji.class.a=testclass1 +test.moji.class.b=testclass2 +test.moji.key.prefix=lcmfi-test- \ No newline at end of file diff --git a/src/test/script/init-mogile-test-data.sh b/src/test/script/init-mogile-test-data.sh new file mode 100755 index 0000000..58c04f9 --- /dev/null +++ b/src/test/script/init-mogile-test-data.sh @@ -0,0 +1,53 @@ +#!/bin/bash +TRACKERS=${moji.tracker.hosts} +DOMAIN=${moji.domain} +KEY_PREFIX=${test.moji.key.prefix} +CLASS=${test.moji.class.a} +IFS=$'\n'; + +function clear_test_data { + KEYS=`mogtool listkey $KEY_PREFIX --trackers=$TRACKERS --domain=$DOMAIN` + + for key in $KEYS + do + if [[ $key == *" files found"* ]] + then + break; + fi + echo "Deleting: '$key' ..." + mogtool delete $key --trackers=$TRACKERS --domain=$DOMAIN + done; +} + +function upload_new_random_file { + head /dev/urandom | uuencode -m - | mogupload --trackers=$TRACKERS --domain=$DOMAIN --key="$KEY_PREFIX$1" --class=$CLASS --file="-" + echo "Created mogile file: '$KEY_PREFIX$1'" +} + +function upload_file { + mogupload --trackers=$TRACKERS --domain=$DOMAIN --key="$KEY_PREFIX$1" --class=$CLASS --file="$2" + echo "Created mogile file: '$KEY_PREFIX$1'" +} + +if [ -z "$KEY_PREFIX" ]; then + echo "You MUST declare a key prefix" + exit 1; +fi + +clear_test_data +upload_new_random_file overwriteThenReadBack +upload_new_random_file exists +upload_new_random_file notExistsAfterDelete +upload_new_random_file rename +upload_new_random_file renameExistingKey1 +upload_new_random_file renameExistingKey2 +upload_new_random_file updateStorageClass +upload_new_random_file updateStorageClassToUnknown +upload_new_random_file list1 +upload_new_random_file list2 +upload_new_random_file list3 +upload_new_random_file getPaths + +upload_file fileOfKnownSize data/fileOfKnownSize.dat +upload_file mogileFileCopyToFile data/mogileFileCopyToFile.dat +exit 0 diff --git a/src/test/script/permission-change.sh b/src/test/script/permission-change.sh new file mode 100755 index 0000000..d634156 --- /dev/null +++ b/src/test/script/permission-change.sh @@ -0,0 +1,3 @@ +#!/bin/bash +chmod u+x target/classes/init-mogile-test-data.sh +echo Changed permission \ No newline at end of file diff --git a/target/classes/fm/last/moji/Moji.class b/target/classes/fm/last/moji/Moji.class new file mode 100644 index 0000000..02db5b8 Binary files /dev/null and b/target/classes/fm/last/moji/Moji.class differ diff --git a/target/classes/fm/last/moji/MojiFactory.class b/target/classes/fm/last/moji/MojiFactory.class new file mode 100644 index 0000000..23b5f0c Binary files /dev/null and b/target/classes/fm/last/moji/MojiFactory.class differ diff --git a/target/classes/fm/last/moji/MojiFile.class b/target/classes/fm/last/moji/MojiFile.class new file mode 100644 index 0000000..4d8a8e8 Binary files /dev/null and b/target/classes/fm/last/moji/MojiFile.class differ diff --git a/target/classes/fm/last/moji/impl/DefaultMojiFactory.class b/target/classes/fm/last/moji/impl/DefaultMojiFactory.class new file mode 100644 index 0000000..7b35fd8 Binary files /dev/null and b/target/classes/fm/last/moji/impl/DefaultMojiFactory.class differ diff --git a/target/classes/fm/last/moji/impl/DeleteCommand.class b/target/classes/fm/last/moji/impl/DeleteCommand.class new file mode 100644 index 0000000..4434ca5 Binary files /dev/null and b/target/classes/fm/last/moji/impl/DeleteCommand.class differ diff --git a/target/classes/fm/last/moji/impl/Executor.class b/target/classes/fm/last/moji/impl/Executor.class new file mode 100644 index 0000000..cc64b1e Binary files /dev/null and b/target/classes/fm/last/moji/impl/Executor.class differ diff --git a/target/classes/fm/last/moji/impl/ExistsCommand.class b/target/classes/fm/last/moji/impl/ExistsCommand.class new file mode 100644 index 0000000..4a24db5 Binary files /dev/null and b/target/classes/fm/last/moji/impl/ExistsCommand.class differ diff --git a/target/classes/fm/last/moji/impl/FileDownloadInputStream.class b/target/classes/fm/last/moji/impl/FileDownloadInputStream.class new file mode 100644 index 0000000..83d9a5f Binary files /dev/null and b/target/classes/fm/last/moji/impl/FileDownloadInputStream.class differ diff --git a/target/classes/fm/last/moji/impl/FileLengthCommand.class b/target/classes/fm/last/moji/impl/FileLengthCommand.class new file mode 100644 index 0000000..da3bb72 Binary files /dev/null and b/target/classes/fm/last/moji/impl/FileLengthCommand.class differ diff --git a/target/classes/fm/last/moji/impl/FileUploadOutputStream.class b/target/classes/fm/last/moji/impl/FileUploadOutputStream.class new file mode 100644 index 0000000..e223922 Binary files /dev/null and b/target/classes/fm/last/moji/impl/FileUploadOutputStream.class differ diff --git a/target/classes/fm/last/moji/impl/GetInputStreamCommand.class b/target/classes/fm/last/moji/impl/GetInputStreamCommand.class new file mode 100644 index 0000000..2bb1278 Binary files /dev/null and b/target/classes/fm/last/moji/impl/GetInputStreamCommand.class differ diff --git a/target/classes/fm/last/moji/impl/GetOutputStreamCommand.class b/target/classes/fm/last/moji/impl/GetOutputStreamCommand.class new file mode 100644 index 0000000..8065d68 Binary files /dev/null and b/target/classes/fm/last/moji/impl/GetOutputStreamCommand.class differ diff --git a/target/classes/fm/last/moji/impl/GetPathsCommand.class b/target/classes/fm/last/moji/impl/GetPathsCommand.class new file mode 100644 index 0000000..f7a1124 Binary files /dev/null and b/target/classes/fm/last/moji/impl/GetPathsCommand.class differ diff --git a/target/classes/fm/last/moji/impl/HttpConnectionFactory.class b/target/classes/fm/last/moji/impl/HttpConnectionFactory.class new file mode 100644 index 0000000..f1404d8 Binary files /dev/null and b/target/classes/fm/last/moji/impl/HttpConnectionFactory.class differ diff --git a/target/classes/fm/last/moji/impl/ListFilesCommand.class b/target/classes/fm/last/moji/impl/ListFilesCommand.class new file mode 100644 index 0000000..491adad Binary files /dev/null and b/target/classes/fm/last/moji/impl/ListFilesCommand.class differ diff --git a/target/classes/fm/last/moji/impl/MojiCommand.class b/target/classes/fm/last/moji/impl/MojiCommand.class new file mode 100644 index 0000000..4d27ebe Binary files /dev/null and b/target/classes/fm/last/moji/impl/MojiCommand.class differ diff --git a/target/classes/fm/last/moji/impl/MojiFileImpl.class b/target/classes/fm/last/moji/impl/MojiFileImpl.class new file mode 100644 index 0000000..f2ccde3 Binary files /dev/null and b/target/classes/fm/last/moji/impl/MojiFileImpl.class differ diff --git a/target/classes/fm/last/moji/impl/MojiImpl.class b/target/classes/fm/last/moji/impl/MojiImpl.class new file mode 100644 index 0000000..35b1b85 Binary files /dev/null and b/target/classes/fm/last/moji/impl/MojiImpl.class differ diff --git a/target/classes/fm/last/moji/impl/PropertyMojiFactory.class b/target/classes/fm/last/moji/impl/PropertyMojiFactory.class new file mode 100644 index 0000000..e8d4fef Binary files /dev/null and b/target/classes/fm/last/moji/impl/PropertyMojiFactory.class differ diff --git a/target/classes/fm/last/moji/impl/RenameCommand.class b/target/classes/fm/last/moji/impl/RenameCommand.class new file mode 100644 index 0000000..0ca466a Binary files /dev/null and b/target/classes/fm/last/moji/impl/RenameCommand.class differ diff --git a/target/classes/fm/last/moji/impl/UpdateStorageClassCommand.class b/target/classes/fm/last/moji/impl/UpdateStorageClassCommand.class new file mode 100644 index 0000000..d69ed77 Binary files /dev/null and b/target/classes/fm/last/moji/impl/UpdateStorageClassCommand.class differ diff --git a/target/classes/fm/last/moji/local/DefaultFileNamingStrategy$KeyPrefixFileNameFilter.class b/target/classes/fm/last/moji/local/DefaultFileNamingStrategy$KeyPrefixFileNameFilter.class new file mode 100644 index 0000000..457c785 Binary files /dev/null and b/target/classes/fm/last/moji/local/DefaultFileNamingStrategy$KeyPrefixFileNameFilter.class differ diff --git a/target/classes/fm/last/moji/local/DefaultFileNamingStrategy.class b/target/classes/fm/last/moji/local/DefaultFileNamingStrategy.class new file mode 100644 index 0000000..4003cc3 Binary files /dev/null and b/target/classes/fm/last/moji/local/DefaultFileNamingStrategy.class differ diff --git a/target/classes/fm/last/moji/local/LocalFileNamingStrategy.class b/target/classes/fm/last/moji/local/LocalFileNamingStrategy.class new file mode 100644 index 0000000..9bb7729 Binary files /dev/null and b/target/classes/fm/last/moji/local/LocalFileNamingStrategy.class differ diff --git a/target/classes/fm/last/moji/local/LocalFileSystemMoji.class b/target/classes/fm/last/moji/local/LocalFileSystemMoji.class new file mode 100644 index 0000000..51ddd93 Binary files /dev/null and b/target/classes/fm/last/moji/local/LocalFileSystemMoji.class differ diff --git a/target/classes/fm/last/moji/local/LocalMojiFile.class b/target/classes/fm/last/moji/local/LocalMojiFile.class new file mode 100644 index 0000000..a50bf06 Binary files /dev/null and b/target/classes/fm/last/moji/local/LocalMojiFile.class differ diff --git a/target/classes/fm/last/moji/spring/SpringMojiBean.class b/target/classes/fm/last/moji/spring/SpringMojiBean.class new file mode 100644 index 0000000..29133c8 Binary files /dev/null and b/target/classes/fm/last/moji/spring/SpringMojiBean.class differ diff --git a/target/classes/fm/last/moji/tracker/Destination.class b/target/classes/fm/last/moji/tracker/Destination.class new file mode 100644 index 0000000..d2eefd7 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/Destination.class differ diff --git a/target/classes/fm/last/moji/tracker/KeyExistsAlreadyException.class b/target/classes/fm/last/moji/tracker/KeyExistsAlreadyException.class new file mode 100644 index 0000000..5127a76 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/KeyExistsAlreadyException.class differ diff --git a/target/classes/fm/last/moji/tracker/Tracker.class b/target/classes/fm/last/moji/tracker/Tracker.class new file mode 100644 index 0000000..7a348a6 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/Tracker.class differ diff --git a/target/classes/fm/last/moji/tracker/TrackerException.class b/target/classes/fm/last/moji/tracker/TrackerException.class new file mode 100644 index 0000000..2a369c2 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/TrackerException.class differ diff --git a/target/classes/fm/last/moji/tracker/TrackerFactory.class b/target/classes/fm/last/moji/tracker/TrackerFactory.class new file mode 100644 index 0000000..72a2073 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/TrackerFactory.class differ diff --git a/target/classes/fm/last/moji/tracker/UnknownKeyException.class b/target/classes/fm/last/moji/tracker/UnknownKeyException.class new file mode 100644 index 0000000..6c96df2 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/UnknownKeyException.class differ diff --git a/target/classes/fm/last/moji/tracker/UnknownStorageClassException.class b/target/classes/fm/last/moji/tracker/UnknownStorageClassException.class new file mode 100644 index 0000000..a68b2e9 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/UnknownStorageClassException.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/AbstractTrackerFactory.class b/target/classes/fm/last/moji/tracker/impl/AbstractTrackerFactory.class new file mode 100644 index 0000000..553a39b Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/AbstractTrackerFactory.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/Charsets.class b/target/classes/fm/last/moji/tracker/impl/Charsets.class new file mode 100644 index 0000000..6126f54 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/Charsets.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/CommunicationException.class b/target/classes/fm/last/moji/tracker/impl/CommunicationException.class new file mode 100644 index 0000000..00e7676 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/CommunicationException.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/CreateOpenOperation.class b/target/classes/fm/last/moji/tracker/impl/CreateOpenOperation.class new file mode 100644 index 0000000..ed887a7 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/CreateOpenOperation.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/ErrorCode.class b/target/classes/fm/last/moji/tracker/impl/ErrorCode.class new file mode 100644 index 0000000..a20215a Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/ErrorCode.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/GetPathsOperation.class b/target/classes/fm/last/moji/tracker/impl/GetPathsOperation.class new file mode 100644 index 0000000..ef7e2c0 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/GetPathsOperation.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/InetSocketAddressFactory.class b/target/classes/fm/last/moji/tracker/impl/InetSocketAddressFactory.class new file mode 100644 index 0000000..4e0f840 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/InetSocketAddressFactory.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/ListKeysOperation.class b/target/classes/fm/last/moji/tracker/impl/ListKeysOperation.class new file mode 100644 index 0000000..a148ae6 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/ListKeysOperation.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/Request$Builder.class b/target/classes/fm/last/moji/tracker/impl/Request$Builder.class new file mode 100644 index 0000000..c014d5a Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/Request$Builder.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/Request.class b/target/classes/fm/last/moji/tracker/impl/Request.class new file mode 100644 index 0000000..7d2d81f Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/Request.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/RequestHandler.class b/target/classes/fm/last/moji/tracker/impl/RequestHandler.class new file mode 100644 index 0000000..ad5d72a Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/RequestHandler.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/Response.class b/target/classes/fm/last/moji/tracker/impl/Response.class new file mode 100644 index 0000000..3dd3001 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/Response.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/ResponseStatus.class b/target/classes/fm/last/moji/tracker/impl/ResponseStatus.class new file mode 100644 index 0000000..1672678 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/ResponseStatus.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/SingleHostTrackerFactory.class b/target/classes/fm/last/moji/tracker/impl/SingleHostTrackerFactory.class new file mode 100644 index 0000000..c920354 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/SingleHostTrackerFactory.class differ diff --git a/target/classes/fm/last/moji/tracker/impl/TrackerImpl.class b/target/classes/fm/last/moji/tracker/impl/TrackerImpl.class new file mode 100644 index 0000000..47ad49a Binary files /dev/null and b/target/classes/fm/last/moji/tracker/impl/TrackerImpl.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/BorrowedTracker.class b/target/classes/fm/last/moji/tracker/pool/BorrowedTracker.class new file mode 100644 index 0000000..f790f97 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/BorrowedTracker.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/HostPriorityComparator.class b/target/classes/fm/last/moji/tracker/pool/HostPriorityComparator.class new file mode 100644 index 0000000..66835da Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/HostPriorityComparator.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost$ResetTask.class b/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost$ResetTask.class new file mode 100644 index 0000000..8385738 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost$ResetTask.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost.class b/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost.class new file mode 100644 index 0000000..5dfa64c Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/ManagedTrackerHost.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool$BorrowedTrackerObjectPoolFactory.class b/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool$BorrowedTrackerObjectPoolFactory.class new file mode 100644 index 0000000..1350219 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool$BorrowedTrackerObjectPoolFactory.class differ diff --git a/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool.class b/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool.class new file mode 100644 index 0000000..06cbf70 Binary files /dev/null and b/target/classes/fm/last/moji/tracker/pool/MultiHostTrackerPool.class differ diff --git a/target/classes/init-mogile-test-data.sh b/target/classes/init-mogile-test-data.sh new file mode 100644 index 0000000..4fca5d5 --- /dev/null +++ b/target/classes/init-mogile-test-data.sh @@ -0,0 +1,53 @@ +#!/bin/bash +TRACKERS=localhost:7001 +DOMAIN=testdomain +KEY_PREFIX=lcmfi-test- +CLASS=testclass1 +IFS=$'\n'; + +function clear_test_data { + KEYS=`mogtool listkey $KEY_PREFIX --trackers=$TRACKERS --domain=$DOMAIN` + + for key in $KEYS + do + if [[ $key == *" files found"* ]] + then + break; + fi + echo "Deleting: '$key' ..." + mogtool delete $key --trackers=$TRACKERS --domain=$DOMAIN + done; +} + +function upload_new_random_file { + head /dev/urandom | uuencode -m - | mogupload --trackers=$TRACKERS --domain=$DOMAIN --key="$KEY_PREFIX$1" --class=$CLASS --file="-" + echo "Created mogile file: '$KEY_PREFIX$1'" +} + +function upload_file { + mogupload --trackers=$TRACKERS --domain=$DOMAIN --key="$KEY_PREFIX$1" --class=$CLASS --file="$2" + echo "Created mogile file: '$KEY_PREFIX$1'" +} + +if [ -z "$KEY_PREFIX" ]; then + echo "You MUST declare a key prefix" + exit 1; +fi + +clear_test_data +upload_new_random_file overwriteThenReadBack +upload_new_random_file exists +upload_new_random_file notExistsAfterDelete +upload_new_random_file rename +upload_new_random_file renameExistingKey1 +upload_new_random_file renameExistingKey2 +upload_new_random_file updateStorageClass +upload_new_random_file updateStorageClassToUnknown +upload_new_random_file list1 +upload_new_random_file list2 +upload_new_random_file list3 +upload_new_random_file getPaths + +upload_file fileOfKnownSize data/fileOfKnownSize.dat +upload_file mogileFileCopyToFile data/mogileFileCopyToFile.dat +exit 0 diff --git a/target/classes/permission-change.sh b/target/classes/permission-change.sh new file mode 100644 index 0000000..d634156 --- /dev/null +++ b/target/classes/permission-change.sh @@ -0,0 +1,3 @@ +#!/bin/bash +chmod u+x target/classes/init-mogile-test-data.sh +echo Changed permission \ No newline at end of file diff --git a/target/classes/spring/moji-context.xml b/target/classes/spring/moji-context.xml new file mode 100644 index 0000000..2f25d06 --- /dev/null +++ b/target/classes/spring/moji-context.xml @@ -0,0 +1,18 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/target/test-classes/checkstyle.xml b/target/test-classes/checkstyle.xml new file mode 100644 index 0000000..5a1c437 --- /dev/null +++ b/target/test-classes/checkstyle.xml @@ -0,0 +1,142 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/target/test-classes/findbugsExclude.xml b/target/test-classes/findbugsExclude.xml new file mode 100644 index 0000000..17344f8 --- /dev/null +++ b/target/test-classes/findbugsExclude.xml @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/target/test-classes/fm/last/moji/FakeMogileFsServer$Builder.class b/target/test-classes/fm/last/moji/FakeMogileFsServer$Builder.class new file mode 100644 index 0000000..71564d4 Binary files /dev/null and b/target/test-classes/fm/last/moji/FakeMogileFsServer$Builder.class differ diff --git a/target/test-classes/fm/last/moji/FakeMogileFsServer$Stanza.class b/target/test-classes/fm/last/moji/FakeMogileFsServer$Stanza.class new file mode 100644 index 0000000..32f7a0f Binary files /dev/null and b/target/test-classes/fm/last/moji/FakeMogileFsServer$Stanza.class differ diff --git a/target/test-classes/fm/last/moji/FakeMogileFsServer$TrackerServer.class b/target/test-classes/fm/last/moji/FakeMogileFsServer$TrackerServer.class new file mode 100644 index 0000000..8eacce7 Binary files /dev/null and b/target/test-classes/fm/last/moji/FakeMogileFsServer$TrackerServer.class differ diff --git a/target/test-classes/fm/last/moji/FakeMogileFsServer.class b/target/test-classes/fm/last/moji/FakeMogileFsServer.class new file mode 100644 index 0000000..e720664 Binary files /dev/null and b/target/test-classes/fm/last/moji/FakeMogileFsServer.class differ diff --git a/target/test-classes/fm/last/moji/MojiInstantiationTest.class b/target/test-classes/fm/last/moji/MojiInstantiationTest.class new file mode 100644 index 0000000..68df41b Binary files /dev/null and b/target/test-classes/fm/last/moji/MojiInstantiationTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/DeleteCommandTest.class b/target/test-classes/fm/last/moji/impl/DeleteCommandTest.class new file mode 100644 index 0000000..4992f69 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/DeleteCommandTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/ExecutorTest.class b/target/test-classes/fm/last/moji/impl/ExecutorTest.class new file mode 100644 index 0000000..1c9a8da Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/ExecutorTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/ExistsCommandTest.class b/target/test-classes/fm/last/moji/impl/ExistsCommandTest.class new file mode 100644 index 0000000..8f3bd76 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/ExistsCommandTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/FileDownloadInputStreamTest.class b/target/test-classes/fm/last/moji/impl/FileDownloadInputStreamTest.class new file mode 100644 index 0000000..304fe73 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/FileDownloadInputStreamTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/FileUploadOutputStreamTest.class b/target/test-classes/fm/last/moji/impl/FileUploadOutputStreamTest.class new file mode 100644 index 0000000..a086894 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/FileUploadOutputStreamTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/HttpConnectionFactoryTest.class b/target/test-classes/fm/last/moji/impl/HttpConnectionFactoryTest.class new file mode 100644 index 0000000..f9a5e4b Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/HttpConnectionFactoryTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/ListFilesCommandTest$1.class b/target/test-classes/fm/last/moji/impl/ListFilesCommandTest$1.class new file mode 100644 index 0000000..d1935b0 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/ListFilesCommandTest$1.class differ diff --git a/target/test-classes/fm/last/moji/impl/ListFilesCommandTest.class b/target/test-classes/fm/last/moji/impl/ListFilesCommandTest.class new file mode 100644 index 0000000..102843f Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/ListFilesCommandTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/MojiFileImplTest.class b/target/test-classes/fm/last/moji/impl/MojiFileImplTest.class new file mode 100644 index 0000000..13d7b81 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/MojiFileImplTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/RenameCommandTest.class b/target/test-classes/fm/last/moji/impl/RenameCommandTest.class new file mode 100644 index 0000000..d1ab010 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/RenameCommandTest.class differ diff --git a/target/test-classes/fm/last/moji/impl/UpdateStorageClassCommandTest.class b/target/test-classes/fm/last/moji/impl/UpdateStorageClassCommandTest.class new file mode 100644 index 0000000..d3e38f5 Binary files /dev/null and b/target/test-classes/fm/last/moji/impl/UpdateStorageClassCommandTest.class differ diff --git a/target/test-classes/fm/last/moji/integration/AbstractMojiIT.class b/target/test-classes/fm/last/moji/integration/AbstractMojiIT.class new file mode 100644 index 0000000..08a0f85 Binary files /dev/null and b/target/test-classes/fm/last/moji/integration/AbstractMojiIT.class differ diff --git a/target/test-classes/fm/last/moji/integration/MojiFileIT.class b/target/test-classes/fm/last/moji/integration/MojiFileIT.class new file mode 100644 index 0000000..40d36e6 Binary files /dev/null and b/target/test-classes/fm/last/moji/integration/MojiFileIT.class differ diff --git a/target/test-classes/fm/last/moji/integration/MojiIT.class b/target/test-classes/fm/last/moji/integration/MojiIT.class new file mode 100644 index 0000000..290d35a Binary files /dev/null and b/target/test-classes/fm/last/moji/integration/MojiIT.class differ diff --git a/target/test-classes/fm/last/moji/local/DefaultFileNamingStrategyTest.class b/target/test-classes/fm/last/moji/local/DefaultFileNamingStrategyTest.class new file mode 100644 index 0000000..0311ddc Binary files /dev/null and b/target/test-classes/fm/last/moji/local/DefaultFileNamingStrategyTest.class differ diff --git a/target/test-classes/fm/last/moji/spring/SpringMojiBeanInstantiationTest.class b/target/test-classes/fm/last/moji/spring/SpringMojiBeanInstantiationTest.class new file mode 100644 index 0000000..245e3d3 Binary files /dev/null and b/target/test-classes/fm/last/moji/spring/SpringMojiBeanInstantiationTest.class differ diff --git a/target/test-classes/fm/last/moji/spring/SpringMojiBeanTest.class b/target/test-classes/fm/last/moji/spring/SpringMojiBeanTest.class new file mode 100644 index 0000000..ec87d0a Binary files /dev/null and b/target/test-classes/fm/last/moji/spring/SpringMojiBeanTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/CreateOpenOperationTest.class b/target/test-classes/fm/last/moji/tracker/impl/CreateOpenOperationTest.class new file mode 100644 index 0000000..e57e5e9 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/CreateOpenOperationTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/GetPathsOperationTest.class b/target/test-classes/fm/last/moji/tracker/impl/GetPathsOperationTest.class new file mode 100644 index 0000000..71a99dc Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/GetPathsOperationTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.class b/target/test-classes/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.class new file mode 100644 index 0000000..99cbb74 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/InetSocketAddressFactoryTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest$TrueMatcher.class b/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest$TrueMatcher.class new file mode 100644 index 0000000..ed1c026 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest$TrueMatcher.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest.class b/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest.class new file mode 100644 index 0000000..234019c Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/RequestHandlerTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/RequestTest.class b/target/test-classes/fm/last/moji/tracker/impl/RequestTest.class new file mode 100644 index 0000000..7bfd5da Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/RequestTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/ResponseTest.class b/target/test-classes/fm/last/moji/tracker/impl/ResponseTest.class new file mode 100644 index 0000000..9a2b2f9 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/ResponseTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.class b/target/test-classes/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.class new file mode 100644 index 0000000..0449db5 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/SingleHostTrackerFactoryTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/impl/TrackerImplTest.class b/target/test-classes/fm/last/moji/tracker/impl/TrackerImplTest.class new file mode 100644 index 0000000..a3cd7e0 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/impl/TrackerImplTest.class differ diff --git a/target/test-classes/fm/last/moji/tracker/pool/BorrowedTrackerTest.class b/target/test-classes/fm/last/moji/tracker/pool/BorrowedTrackerTest.class new file mode 100644 index 0000000..c7c5b89 Binary files /dev/null and b/target/test-classes/fm/last/moji/tracker/pool/BorrowedTrackerTest.class differ diff --git a/target/test-classes/log4j.properties b/target/test-classes/log4j.properties new file mode 100644 index 0000000..d7f2348 --- /dev/null +++ b/target/test-classes/log4j.properties @@ -0,0 +1,8 @@ +log4j.rootLogger=INFO, stdout + +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.Target=System.out +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %5p %c{2}:%L - %m%n + +log4j.logger.fm.last = DEBUG \ No newline at end of file diff --git a/target/test-classes/moji.properties b/target/test-classes/moji.properties new file mode 100644 index 0000000..daa6075 --- /dev/null +++ b/target/test-classes/moji.properties @@ -0,0 +1,7 @@ +# My local properties +moji.tracker.hosts=localhost:7001 +moji.domain=testdomain + +test.moji.class.a=testclass1 +test.moji.class.b=testclass2 +test.moji.key.prefix=lcmfi-test- \ No newline at end of file