Retired

Reason for Retirement: Please confirm if this is still a problem in the latest
version of Smart.

Please reopen this issue in the new bugtracker if it is still an issue.
New Bugtracker : http://bugs.launchpad.net/smart
further details:
https://blueprints.launchpad.net/smart/+spec/bug-reporting-migration.

Recently gave smart a try again... on SUSE 10.1 this time.
Everything works except... no proxy support.
Is there any update on this?
How hard would the conversion to urllib2 be?

'urllib' doesn't work with authenticated proxies, while 'urllib2' does.

About the pycurl package, it's up to the distro to decide what to do. It's not
up to Smart to choose one or other lib based on specific distro issues. I'm sure
you'd understand that.

I have quite a bit of experience with python, perhaps I should take a stab at
it. I know that either urllib or urllib2 support user auth (although I think
*both* of them do, but only one does "out-of-the-box")

pycurl would be a problem for suse users because one has to go to packman to get
pycurl (although with opensuse it may be possible to get them to include it,
curl is very cool)

We've already detected some problems with the current fetcher implementation
that uses 'urllib'. For example, it can't handle authenticated proxies
(issue107) so in this case the user is forced to have pycurl.

We're still discussing if the best way to solve those two kinds of problems
would be enforcing pycurl or changing to 'urllib2'.

Because of that, please, continue using pycurl. I'm deferring this ticket for
future review.

Yes - Using FTP with a proxy and Not using Curl does not work. HTTP
and HTTPS with or without Curl seems to work.

The reason (apparently) is that FTP in Smart is handled by ftplib.
Http and Https are handled by urllib. If curl is installed, curl
handles ftp, http and https. From some google searches it seems that
ftplib doesn't support proxies.

It looks like the code once used urllib to do FTP, but that has been
commented out.

If curl is not working for you, or you don't want to use it, you may
want to see if enabling urllib for FTP works. Unfortunately, the only
way I know how to do this is by editing the code (fetcher.py). Just
disable the FTPLIBHandler, and enable "ftp" for the URLLIBHandler.

We are using Curl with Smart at my company because it provides more
features (primarily authentication mechanisms with HTTPS) and I
honestly trust that part of the code more than URLLIBHandler. I think
Smart is better off sticking to just the Curl handler instead of
FTPLIBHandler, URLLIBHandler and URLLIB2Handler...

This should be really easy to test.

1. remove/uninstall/whatever pycurl
2. use an ftp proxy
3. export FTP_PROXY=http://127.0.0.1:3128/  <- easy to see
4. START A TCPDUMP: tcpdump -i lo port 3128
5. run smart update

If you do not see packets going to 3128 (which will of course fail), then it's
not using the proxy.

OK, I've performed some more testing, and it seems that FTP Proxying
via curl with smart works for me. Note that I don't have a real ftp
proxy, but I believe this fake information shows that FTP proxying
should work. I know for certain that HTTP and HTTPS proxying works for
me, as I've mentioned before.

--------------------------------------------------------------------------------------------

[zgarner@test1.emap] 387 export FTP_PROXY=http://somerandomhost.com
[zgarner@test1.emap] 388 sudo smart update
Loading cache...
Updating cache...                                            
#####################################################################################
[100%]

Fetching information for 'Production'...
-> ftp://ops.awarix.com/repository/production/1-awarix/base/release
release                                                               

      [  0%]
error: Failed acquiring information for 'Production':
error: ftp://ops.awarix.com/repository/production/1-awarix/base/release:
Couldn't resolve proxy 'somerandomhost.com'

--------------------------------------------------------------------------------------------

I am using a slightly modified (mostly to fix 'upgrade' bugs) build of
smart on a SuSE Enterprise 9 system.

I'm not sure how to help you further with this, since I cannot
reproduce the problem. Maybe some of the official Smart developers can
help you further.

If you are desperate however, you could edit 'smart/fetcher.py',
around line 1619, in the PyCurlHandler class. Add something like:
      proxyfile = "/etc/smart/proxy";
      if os.path.isfile(proxyfile):
          handle.setopt(pycurl.CURLOPT_PROXY, os.open(proxyfile).readline));

"pycurl.CURLOPT_PROXY" is equivalent to the "--proxy" option when
running curl. This would manually set the proxy for smart/curl to the
first line of "/etc/smart/proxy", if it exists. It probably wouldn't
be a bad idea to print out a debug statement to ensure that the
PyCurlHandler is actually being used.

At per message 384, I installed python-pycurl.

curl from the commandline ALSO uses the proxy.

In summary, for ftp mirrors:

smart without pycurl:  no proxy
curl or smart with pycurl:  proxy

Did you say you were using Smart with Curl? If so, does the FTP_PROXY
variable work with plain curl?

On 2/19/06, Jon at Labix Tracker <tracker@labix.org> wrote:
>
> Jon <jnelson-smartpm@jamponi.net> added the comment:
>
> I don't think any of my mirrors use http - they all use ftp.  And ftp doesn't
> seem to be proxied.  I will note that I do have ftp_proxy and FTP_PROXY set (and
> both have the same value), but smart seems to ignore it.
>
> _______________________________________
> Labix issue tracker <tracker@labix.org>
> <http://tracker.labix.org/issue123>
> _______________________________________
>

I don't think any of my mirrors use http - they all use ftp.  And ftp doesn't
seem to be proxied.  I will note that I do have ftp_proxy and FTP_PROXY set (and
both have the same value), but smart seems to ignore it.

HTTP and HTTPS proxy support works for me with environment variables:
"http_proxy" and "HTTPS_PROXY" (note the capitalization) using Curl
with Smart. I haven't tried FTP.

On 2/19/06, Jon at Labix Tracker <tracker@labix.org> wrote:
>
> Jon <jnelson-smartpm@jamponi.net> added the comment:
>
> I installed python-pycurl and that did the trick (note that python-pycurl is
> only available from packman, it is not available from SUSE).
>
> The proxy is never contacted, I performed a tcpdump and see the request go (or
> try to go, it's blocked by the firewall) to the origin server instead of going
> through the proxy.
>
> However, IMO, requiring python-pycurl doesn't seem right - the stock python
> standard library urllib will use proxies as is - is smart not using them?
>
> _______________________________________
> Labix issue tracker <tracker@labix.org>
> <http://tracker.labix.org/issue123>
> _______________________________________
>

I installed python-pycurl and that did the trick (note that python-pycurl is
only available from packman, it is not available from SUSE).

The proxy is never contacted, I performed a tcpdump and see the request go (or
try to go, it's blocked by the firewall) to the origin server instead of going
through the proxy.

However, IMO, requiring python-pycurl doesn't seem right - the stock python
standard library urllib will use proxies as is - is smart not using them?

Could you paste here the error message you get from Smart?

If you have access to your proxy logs, can you send any relevant messages?

At last, can you try with 'pycurl' installed?

it would appear that smart can't do ftp over a proxy, despite environment
variables being set, etc... it may also be true for http.