Details
-
Type:
New Feature
-
Status:
Resolved
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: 0.20.2
-
Fix Version/s: None
-
Component/s: fs
-
Labels:None
-
Release Note:Adds support for Parascale storage.
Description
Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters. We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?
-
- HADOOP-6704_0_20_2-2.patch
- 06/May/10 04:08
- 160 kB
- Neil Bliss
-
- HADOOP-6704_0_20_2-3.patch
- 21/May/10 17:03
- 163 kB
- Neil Bliss
-
- HADOOP-6704.0.20.2.patch
- 29/Apr/10 16:56
- 158 kB
- Neil Bliss
-
- HADOOP-6704.patch
- 29/Apr/10 16:56
- 126 kB
- Neil Bliss
-
- HADOOP-6704-10.patch
- 21/May/10 11:03
- 129 kB
- Neil Bliss
-
- HADOOP-6704-2.patch
- 30/Apr/10 23:22
- 126 kB
- Neil Bliss
-
- HADOOP-6704-3.patch
- 04/May/10 15:49
- 126 kB
- Neil Bliss
-
- HADOOP-6704-4.patch
- 04/May/10 17:19
- 126 kB
- Neil Bliss
-
- HADOOP-6704-5.patch
- 04/May/10 22:55
- 126 kB
- Neil Bliss
-
- HADOOP-6704-6.patch
- 05/May/10 00:01
- 126 kB
- Neil Bliss
-
- HADOOP-6704-7.patch
- 05/May/10 05:23
- 126 kB
- Neil Bliss
-
- HADOOP-6704-8.patch
- 07/May/10 00:02
- 129 kB
- Neil Bliss
Activity
- All
- Comments
- Work Log
- History
- Activity
- Subversion Commits
- This is a new feature, so the patch should be against SVN_HEAD, not an older version of Hadoop.
- The newer version of Hadoop is moving to some new FS APIs; you might want to consider working with them.
HDFS-708has discussed the issue of stress testing filesystems; this new filesystem back end could be a use case, if you are willing to participate.- One issue with all third party filesystems is regression testing: they don't get enough of it. If there is any way to make this easier -and that could include you running a local version of Hudson to grab SVN_HEAD of Hadoop and testing MR jobs over your filestore, then end users will be grateful.
- Involvement in testing forthcoming releases is equally important, as is ongoing maintenance. It is really hard for an OSS project to test/maintain code that works with other peoples infrastructure, and motivation can be trouble too, so you have to be willing to stay involved -otherwise the code just gradually stops working.
- This is a new feature, so the patch should be against SVN_HEAD, not an older version of Hadoop.
- The newer version of Hadoop is moving to some new FS APIs; you might want to consider working with them.
HDFS-708has discussed the issue of stress testing filesystems; this new filesystem back end could be a use case, if you are willing to participate.- One issue with all third party filesystems is regression testing: they don't get enough of it. If there is any way to make this easier -and that could include you running a local version of Hudson to grab SVN_HEAD of Hadoop and testing MR jobs over your filestore, then end users will be grateful.
- Involvement in testing forthcoming releases is equally important, as is ongoing maintenance. It is really hard for an OSS project to test/maintain code that works with other peoples infrastructure, and motivation can be trouble too, so you have to be willing to stay involved -otherwise the code just gradually stops working.
Here's a patch against the 0.20.2 tag. A patch against trunk will be forthcoming shortly.
I'll take a look at the stress testing Jira and see if it's something we can help with.
that patch still contains unit test failures. I will fix it and respin the patch.
Patches to -0.20.2 and -trunk adding support for Parascale storage as the DFS.
Adds support for Parascale storage as the backing store for Hadoop instances.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443204/HADOOP-6704.patch
against trunk revision 939510.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 61 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1017 warnings).
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
-1 release audit. The applied patch generated 5 release audit warnings (more than the trunk's current 1 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443345/HADOOP-6704-2.patch
against trunk revision 939864.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 61 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).
-1 findbugs. The patch appears to introduce 3 new Findbugs warnings.
-1 release audit. The applied patch generated 5 release audit warnings (more than the trunk's current 1 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/64/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/64/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/64/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/64/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/64/console
This message is automatically generated.
The -3 version of this patch addresses findbugs, releaseaudit, and checkstyle warnings that were present in the -2 version.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443588/HADOOP-6704-3.patch
against trunk revision 940527.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/498/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/498/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/498/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/498/console
This message is automatically generated.
I now see the problem resulting in the findbugs warning. Fixing it now.
-3 patch still contains a findbugs warning. This will be addressed in the -4 version of the patch.
The -4 version of this patch addresses the findbugs warning from the -3 version, specifically it adds a ChunkInfo.equals() implementation.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443598/HADOOP-6704-4.patch
against trunk revision 940527.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/499/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/499/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/499/console
This message is automatically generated.
I ran findbugs locally before submitting the -4 version of the patch, and it came back clean. I can't help but wonder why I'm seeing different results from Hudson. The findbugs complaint is now that I don't have ChunkInfo.hashCode() implemented. I'll implement that, and resubmit to see what Hudson has to say then.
ahhh, findbugs is an external dependency that's not installed on my system. Running "ant findbugs" without having it installed doesn't complain that it's not there, so I didn't know I was missing the tool.
Version -5 of this patch. Addresses all found findbugs and checkstyle warnings. Adds ChunkInfo.hashCode() implementation.
I think I goofed up the order of operations in submitting the -5 version of the patch. I'm going to resubmit it.
Resubmit of the -5 version of this patch. Adds ChunkInfo.hashCode().
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443654/HADOOP-6704-5.patch
against trunk revision 940989.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/501/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/501/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/501/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/501/console
This message is automatically generated.
This latest test failure comes from 5 javac warnings about the use of a deprecated interface to org.apache.hadoop.fs.Path.makeQualified(). The deprecation message in the source indicates to use makeQualified(URI, Path) instead of makeQualified(FileSystem). In looking through the rest of the codebase, however, every reference to makeQualified is using the deprecated interface. How robust is the new interface, and is the old interface really deprecated?
Version -6 of this patch, revises org.apache.hadoop.fs.Path.makeQualified() references to use non-deprecated interfaces.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443664/HADOOP-6704-6.patch
against trunk revision 940989.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1018 javac compiler warnings (more than the trunk's current 1017 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/502/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/502/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/502/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/502/console
This message is automatically generated.
Version -7 of this patch. Remove the deprecated getName() interface from RawParascaleFileSystem.java, and remove the corresponding test case.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443679/HADOOP-6704-7.patch
against trunk revision 940989.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/503/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/503/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/503/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/503/console
This message is automatically generated.
HADOOP-6704_0_20_2-2.patch applies the funtionality of HADOOP-6704-7.patch to the 0.20.2 branch.
We have a few more in-house changes to merge into this patch. Patch version -8 forthcoming shortly.
HADOOP-6704-8.patch cleans up the testing infrastructure slightly and more completely cleans up after test runs.
HADOOP-6704-8.patch incorporates improvements in the parascale unit tests.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443914/HADOOP-6704-8.patch
against trunk revision 941662.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 55 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/509/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/509/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/509/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/509/console
This message is automatically generated.
Someone needs to review the patch, apologies for the slow response time, I think people are pretty swamped with internal stuff and blockers for the 21 release.
I understand completely! I was just checking to make sure there wasn't something I was supposed to be doing at this point. Carry on!
There have been a few bugfixes made on the internal development tree, so I'll withdraw the -8 patch and submit a newer one with the current code.
HADOOP-6704-9.patch enables usage of Parascale storage. The -9 version of this patch contains improvements to the cleanup after unit tests have been run, and an important bugfix in the HTTPChunkLocator code affecting operation in multithreaded client code.
HADOOP-6704-10.patch contains the same fixes described in the (since removed) -9 patch.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445159/HADOOP-6704-10.patch
against trunk revision 946976.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 53 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/537/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/537/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/537/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/537/console
This message is automatically generated.
HADOOP-6704_0_20_2-3.patch provides important bugfixes over the -2 patch for the 0.20.2 branch.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445180/HADOOP-6704_0_20_2-3.patch
against trunk revision 1031422.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 95 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/40//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445180/HADOOP-6704_0_20_2-3.patch
against trunk revision 1071364.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 95 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/276//console
This message is automatically generated.
Closing out as stale, this issue hasn't been updated in over a year.
Seems like it should go in o.a.h.fs with the other non-hdfs file systems if it's going to be actively maintained. Will need some unit tests to help prevent people from breaking it since hudson doesn't have a parascale backend.