Azkaban 3.0 support

[dmvieira]

I'm getting HTTPError: 411 Client Error: Length Required for url

[mtth]

Which command does the error happen for--is it during uploading? Unfortunately I don't have access to an Azkaban server to test a fix; can you try adding a content-length header here, set to 0?

[dmvieira]

When I do build or upload commands. I'll try with It, but why?

[mtth]

It could be that Azkaban switched to a stricter server implementation in 3.0 which requires this header (or somehow doesn't support chunked encoding at all anymore?). Before trying more complex fixes, I was curious whether adding this header and simply setting it to a 0 sentinel value would work.

Also, looking a bit at Azkaban's API docs, it looks like there are now 3 distinct server modes (solo, dual, distributed). Do you know which setup you are talking to?

[dmvieira]

My server is distributed, but using API with curl worked on web server Em sáb, 17 de mar de 2018 12:34, Matthieu Monsch <notifications@github.com> escreveu:

…

It could be that Azkaban switched to a stricter server implementation in 3.0 which requires this header (or somehow doesn't support chunked encoding at all anymore?). Before trying more complex fixes, I was curious whether adding this header and simply setting it to a 0 sentinel value would work. Also, looking a bit at Azkaban's API docs, it looks like there are now 3 distinct server modes <http://azkaban.github.io/azkaban/docs/latest/#getting-started> (solo, dual, distributed). Do you know which setup you are talking to? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDqhRYwT-hF_ZY261Dcfs-7usXr0NPLks5tfS0JgaJpZM4SuQMF> .

[dmvieira]

I did this change and now code is like this:

self.headers = {
  'Content-Length': '0',
  'Content-Type': 'multipart/form-data; boundary=%s' % (self._boundary, )
}

But now there is an error:

TypeError: object of type 'MultipartForm' has no len()

Here the stack trace:

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053, in request
self._send_request(method, url, body, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093, in _send_request
self.endheaders(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 897, in _send_output
self.send(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 869, in send
self.sock.sendall(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 718, in sendall
amount = len(data)
TypeError: object of type 'MultipartForm' has no len()

[dmvieira]

Do you think about change this project to azkaban 3 or can I build one for this?

[mtth]

I'd happily merge a PR to support Azkaban 3 if you submit one.

[trentgerman]

👋 I have a test server set up with Azkaban version 3.33, and I would be willing to help with this. If you can give me any guidance that would be appreciated! I can authenticate with the server, but I get a 400 client error when trying to upload a zip. I will need to investigate the MultipartForm class more before I know why.

Another question - I'm curious about the tests. It appears that many of the tests try to communicate with a real server if 'test.alias' is defined in ~/.azkabanrc, otherwise they are skipped. Is this the case?

Regards,
-Trent

[dmvieira]

We did another project for Azkaban 3 upload and schedule of jobs: https://github.com/globocom/azkaban-cli/

We did another project to make Azkaban 3 flows for V1 or V2 with plugin support: https://github.com/globocom/auror-core