A patch for Cloudera 4.1.2 MRv1

[sebastienc]

Hi,

We've had to implement Mesos with CDH 4.1.2 so I thought I'd give back the patch I've ported.

Regards,

Sebastien Coutu

[vinodkone]

Hey Sebastien, thanks for CDH4 patch! We would love to include this patch, but there are a few things you need to know first.

--> We have recently changed how the Hadoop port works on Mesos (https://reviews.apache.org/r/8965/)

--> In a nutshell, we no longer need to patch Hadoop internals (e.g TaskTrackerInstrumentation.java) and we have considerably changed how MesosScheduler and MesosExecutor works.

--> Also, the "cdh-mesos" repo is no longer actively maintained. The Hadoop support is bundled into the "mesos" repo (https://github.com/apache/mesos).

So, if you could send your CDH4 patch based on the latest "mesos" branch, that would make us easy to review and submit the patch.

Alternatively, if you could tell us what specifically you had to change for CDH4 support (compared to the existing CDH3 support), that would make us understand the changes better.

Thanks,
Vinod

[sebastienc]

Hi Vinod,

Yes, I saw about the new scheduler and I'm planning on trying it on our
test cluster soon. How stable is it?

Yep, I could definitely create a new patch based on the latest version of
the repo. Is a git patch ok? Do you have specific options I should use?

Thanks,

Sebastien
On Jan 30, 2013 6:02 PM, "Vinod Kone" notifications@github.com wrote:

Hey Sebastien, thanks for CDH4 patch! We would love to include this patch,
but there are a few things you need to know first.

--> We have recently changed how the Hadoop port works on Mesos (
https://reviews.apache.org/r/8965/)

--> In a nutshell, we no longer need to patch Hadoop internals (e.g
TaskTrackerInstrumentation.java) and we have considerably changed how
MesosScheduler and MesosExecutor works.

--> Also, the "cdh-mesos" repo is no longer actively maintained. The
Hadoop support is bundled into the "mesos" repo (
https://github.com/apache/mesos).

So, if you could send your CDH4 patch based on the latest "mesos" branch,
that would make us easy to review and submit the patch.

Alternatively, if you could tell us what specifically you had to change
for CDH4 support (compared to the existing CDH3 support), that would make
us understand the changes better.

Thanks,
Vinod

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1#issuecomment-12917747.

[vinodkone]

We committed the latest Hadoop port on trunk, but haven't yet backported it to our stable releases (0.9.0 or 0.10.0).

As for the stability of trunk, I'm not aware of any organizations running it in production. At Twitter, we run Mesos (not Hadoop port) that is based off 0.10.0. Having said that, I've heard folks at AirBnB are planning on testing the new Hadoop port.

Regarding patch, a git patch would be perfect. Please base it off https://github.com/apache/mesos.

[sebastienc]

Cool I'll test it (trunk) with a test cluster we've got handy at Hopper. In
the meantime I'll get you a new patch based on the repo you mentioned.
On Jan 30, 2013 7:46 PM, "Vinod Kone" notifications@github.com wrote:

We committed the latest Hadoop port on trunk, but haven't yet backported
it to our stable releases (0.9.0 or 0.10.0).

As for the stability of trunk, I'm not aware of any organizations running
it in production. At Twitter, we run Mesos (not Hadoop port) that is based
off 0.10.0. Having said that, I've heard folks at AirBnB are planning on
testing the new Hadoop port.

Regarding patch, a git patch would be perfect. Please base it off
https://github.com/apache/mesos.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1#issuecomment-12921455.

[vinodkone]

Just curious. Were you running the CDH4 tarball from
http://archive.cloudera.com/cdh4/cdh/4/mr1-2.0.0-mr1-cdh4.1.2.tar.gz ?

For the life of me, I can't seem to get it to build. I get some unresolved
ivy dependencies (log4j, etc) when running 'ant' on my MacOSX.

-- Vinod

On Wed, Jan 30, 2013 at 4:51 PM, Sebastien Coutu
notifications@github.comwrote:

Cool I'll test it (trunk) with a test cluster we've got handy at Hopper.
In
the meantime I'll get you a new patch based on the repo you mentioned.
On Jan 30, 2013 7:46 PM, "Vinod Kone" notifications@github.com wrote:

We committed the latest Hadoop port on trunk, but haven't yet backported
it to our stable releases (0.9.0 or 0.10.0).

As for the stability of trunk, I'm not aware of any organizations
running
it in production. At Twitter, we run Mesos (not Hadoop port) that is
based
off 0.10.0. Having said that, I've heard folks at AirBnB are planning on
testing the new Hadoop port.

Regarding patch, a git patch would be perfect. Please base it off
https://github.com/apache/mesos.

—
Reply to this email directly or view it on GitHub<
https://github.com/mesos/cdh-mesos/pull/1#issuecomment-12921455>.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1#issuecomment-12921599.

[sebastienc]

Yep, against mr1-2.0.0-mr1-cdh4.1.2.tar.gz. I've built it on a CentOS 6.1
system replacing the SRPMS's mr1-2.0.0-mr1-cdh4.1.2.tar.gz file with my
patched version of it. You'll need to use the cloudera/do-release-build
script to build it. It will apply a few patches and download the
dependencies using Ivy. Just make sure you have the right ant "plugins"
installed along with Forrest in your path. If you need anything, let me
know, I'll try to help :)

On Wed, Jan 30, 2013 at 9:26 PM, Vinod Kone notifications@github.comwrote:

Just curious. Were you running the CDH4 tarball from
http://archive.cloudera.com/cdh4/cdh/4/mr1-2.0.0-mr1-cdh4.1.2.tar.gz ?

For the life of me, I can't seem to get it to build. I get some unresolved
ivy dependencies (log4j, etc) when running 'ant' on my MacOSX.

-- Vinod

On Wed, Jan 30, 2013 at 4:51 PM, Sebastien Coutu
notifications@github.comwrote:

Cool I'll test it (trunk) with a test cluster we've got handy at Hopper.
In
the meantime I'll get you a new patch based on the repo you mentioned.
On Jan 30, 2013 7:46 PM, "Vinod Kone" notifications@github.com wrote:

We committed the latest Hadoop port on trunk, but haven't yet
backported
it to our stable releases (0.9.0 or 0.10.0).

As for the stability of trunk, I'm not aware of any organizations
running
it in production. At Twitter, we run Mesos (not Hadoop port) that is
based
off 0.10.0. Having said that, I've heard folks at AirBnB are planning
on
testing the new Hadoop port.

Regarding patch, a git patch would be perfect. Please base it off
https://github.com/apache/mesos.

—
Reply to this email directly or view it on GitHub<
https://github.com/mesos/cdh-mesos/pull/1#issuecomment-12921455>.

—
Reply to this email directly or view it on GitHub<
https://github.com/mesos/cdh-mesos/pull/1#issuecomment-12921599>.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1#issuecomment-12924608.

[vinodkone]

Thanks for the tip.

https://reviews.apache.org/r/9174/

[sebastienc]

Hi Vinod,

I see the review is now closed, just to be sure, do you still need me to
update the patch based on the new repo?

Thanks,

Sebastien

On Thu, Jan 31, 2013 at 3:26 AM, Vinod Kone notifications@github.comwrote:

Thanks for the tip.

https://reviews.apache.org/r/9174/

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1#issuecomment-12932349.

[vinodkone]

That review (https://reviews.apache.org/r/9174/) is still under review :)

Feel free to take a look at the review and suggest any improvements/fixes based on your experience with CDH4 so far. Also, I've only tested the patch locally. So, if you could test it out on your end, that would be great!