Closed
Bug 1582
Opened 26 years ago
Closed 23 years ago
[NECKO][webshell] Send HTTP Referer field to server, "network.sendRefererHeader"
Categories
(SeaMonkey :: UI Design, defect, P1)
Tracking
(Not tracked)
VERIFIED
FIXED
M15
People
(Reporter: dbaron, Assigned: gagan)
References
()
Details
(Keywords: crash, helpwanted, testcase, Whiteboard: [PDT+] ETA 02/22/00)
The HTTP_REFERER variable is not sent to the server... it should have the URL
of the referring page (and should not be sent if the URL was typed in or if it
is the home page, etc.).
Reporter | ||
Updated•26 years ago
|
Reporter | ||
Comment 2•26 years ago
|
||
Added test page to URL that shows what HTTP fields are being sent:
http://www.fas.harvard.edu/~dbaron/html/httpreq
Comment 3•26 years ago
|
||
setting paulmac as QA contact for all gagan's bugs (sorry for the spam)
Updated•26 years ago
|
Target Milestone: M3 → M4
Comment 4•26 years ago
|
||
Moving to M4.
Per DP's suggestion marking these till M8. Though Necko lands with M7, we will
be able to verify it for M8.
Comment 8•26 years ago
|
||
I'm moving this to target M9, Necko will be enabled somewhere during late M8 or
early M9. We will need to get on this and it cannot be postponed past the M9
milestone.
Comment 10•26 years ago
|
||
Changing all Networking Library/Browser bugs to Networking-Core component for
Browser.
Occasionally, Bugzilla will burp and cause Verified bugs to reopen when I do
this in a bulk change. If this happens, I will fix. ;-)
Assignee | ||
Comment 11•26 years ago
|
||
Neither it is sent with Necko. Jud what was the decision on this regard? Did we
want to set a preference for the same?
Comment 12•26 years ago
|
||
This isn't a clear cut decision. However, we've hashed out a few absolutes wrt
how the referrer should be set.
1. It is _never_ sent when a user types in a url.
2. It is _always_ sent when a link is clicked on (there is no pref to turn this
off).
3. In the 4.5+ (somewhere around there) codebase we did not send the referrer
header when the url was an image (i.e. the image lib wouldn't set referrer
anymore), _unless_ it had been clicked on (i.e. in an <a href=x><img src=></a>
For the time being, it's ok to send it everytime, except when the user types in
a url.
Reporter | ||
Comment 13•26 years ago
|
||
I think you should send the referer for inline images. It allows people to see
where bad links come from. I was under the impression most browsers did so. A
quick look at referer logs shows this is true.
Section 14.36 of HTTP 1.1 draft,
http://www.w3.org/Protocols/HTTP/1.1/draft-ietf-http-v11-spec-rev-06.txt , says:
The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained (the "referrer", although the
header field is misspelled.) The Referer request-header allows a
server to generate lists of back-links to resources for interest,
logging, optimized caching, etc. It also allows obsolete or mistyped
links to be traced for maintenance. The Referer field MUST NOT be
sent if the Request-URI was obtained from a source that does not have
its own URI, such as input from the user keyboard.
Referer = "Referer" ":" ( absoluteURI | relativeURI )
Example:
Referer: http://www.w3.org/hypertext/DataSources/Overview.html
If the field value is a relative URI, it SHOULD be interpreted
relative to the Request-URI. The URI MUST NOT include a fragment. See
section 15.1.3 for security considerations.
Comment 14•26 years ago
|
||
I'm having trouble understanding how sending a referrer w/ every inline image
could help bad links get resolved. Can you please provide an explaination?
Reporter | ||
Comment 15•26 years ago
|
||
If people move inline images, they may not know all the files that refer to
those images - inline images are often used by multiple pages.
Sometimes inline images are even used by people on other sites (and are
sometimes intended to be). If someone else starts using my inline images, I
don't want to see my error logs fill up and not know who is still using those
images (possibly even me!).
This is what the large majority of web browsers do, I think, judging from the
frequency of inline images appearing in the referrer logs I just looked at. I
just went through the last few hundred (?) lines of a referer log for a server
where the root page "/" links to one inline image. There were 28 refers to the
root page of the server. 26 of those 28 were immediately followed by a refer
from the home page to the inline image in it. Many of these hits were probably
from Netscape 4.x. The two times there was no referer for the inline image
could easily be people not loading inline images (or using lynx) or who clicked
on a link in the page before the image started loading.
If you break this behavior, you will make life much harder for web server
admins.
Comment 16•26 years ago
|
||
This is good info. Our motivation for halting referrer fields on inline images
was bandwidth based. we wanted to reduce the size of the HTTP request. However,
it appears we didn't put enough weight on the point you're now making. I guess
we just assumed the log admin receiving the referrer of the broken inline
wouldn't be using this info. Looks like we assumed incorrectly and we should be
sending the referrer header along w/ inline images.
Thanks for clarifying.
Assignee: gagan → don
Status: ASSIGNED → NEW
Component: Networking-Core → XPApps
Assignee | ||
Comment 17•26 years ago
|
||
In light of the discussion we have had here, it looks like the referrer field
decision really will have a tighter association with the UI. And so for more
details on who is going to add this where I am assigning this to webshell. (Does
webshell know the difference between a link clicked or one typed in Location
bar?) Since I dont see webshell in the components list I am assigning this to
XPApps... Feel free to correct me.
Assignee: don → radha
Summary: NECKO: referer not sent → [NECKO] referer not sent
Comment 18•26 years ago
|
||
Radha, do we need to fix this in M9?
Comment 19•26 years ago
|
||
Answering and asking few questions....
1) I read thro' this and I am not sure why this is a UI thing.
2) webshell does know the difference between a link click and a url typed in the
urlbar. But only to some extent. After a couple of calls, they both go thro' the
same codepath down to Necko
3) If necko *really* needs to know whether the url load request occured because
of link click or otherwise, then webshell needs to pass some info to DocLoader.
4) Jevering says that this *has* to be fixed in M9.
5) cc'ing warren to help clarify things.
Comment 20•26 years ago
|
||
I agree that the webshell needs to pass the referrer to the doc loader, but I'm
not sure how this worked before. The doc loader should define a callback that
the protocol (http) can call to get the referrer. Gagan should handle that
part.
Judson was going to look at the web shell code to fix up the default protocol,
www. and .com parts -- I would suspect that the distinction between the
location bar and clicking on a link would be coming into play in that same
general vacinity, so maybe he should handle that part.
Updated•26 years ago
|
Assignee: radha → valeski
Comment 21•26 years ago
|
||
Based on warren's comments, I'm reassigning this to valeski. Let scc know if
you want some API modified in webshell since he is the current owner of it.
Note:
When the url is typed in to the urlbar, BrowserAppcore calls in to webshell
thro' LoadURL(). When a link is clicked, layout calls in to
nsWebShell::HandleLinkClick() which eventually calls LoadUrl().
cc'ing scc
Updated•26 years ago
|
Assignee: valeski → gagan
Assignee | ||
Comment 22•26 years ago
|
||
I recently fixed post requests to send a referer field since most post servers
require referer field. Pls. use the same logic in whereever the inline URLs for
the imageas are being constructed. For more details look at
mozilla/layout/html/forms/nsFormFrame.cpp and grep for "Referer"
Updated•26 years ago
|
Whiteboard: does confusion still reign?
Comment 23•26 years ago
|
||
*** Bug 10099 has been marked as a duplicate of this bug. ***
Updated•26 years ago
|
Status: NEW → ASSIGNED
Target Milestone: M9 → M10
Comment 24•26 years ago
|
||
OK. I spoke to Rick. Here is what we came up with,
0) Since load groups get passed around to all types of necko customers,
it looked like a good centralized place to manage referer.
1) add a baseuri member to load group
2) Based on privacy preference settings(which webshell will know about), and
other deciding factors, webshell/docloader will set the base uri in the load
group.
3) And down in necko, this base uri will be used to set the referer field. The
point is, if the referer has to be sent, baseuri will be set, if not, baseuri
will not be set.
3) Other components like libimage has to do similar things for inline images.
when a image load request comes thro' does libimg know about the base url? maybe
pnunn can answer this.
4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?
adding rpotts,pnunn to cc list. warren, can you comment on this.
I don't believe this is a M9 blocker. So, moving to M10.
Comment 25•26 years ago
|
||
Sounds like a good plan. One question: Should the baseuri be the same thing as
the first URI inserted into the group?
Priority: P2 → P1
Summary: [NECKO] referer not sent → [FEATURE][NECKO] Send HTTP_REFERER value to server
Whiteboard: It's b-a-c-k! and Radha is investigating it ... → Radha is investigating
Summary: [FEATURE][NECKO] Send HTTP_REFERER value to server → [NECKO] Send HTTP_REFERER value to server
Comment 26•26 years ago
|
||
*** Bug 5408 has been marked as a duplicate of this bug. ***
Updated•25 years ago
|
Whiteboard: Radha is investigating → Waiting for input from rpotts
Updated•25 years ago
|
Whiteboard: Waiting for input from rpotts → HELP WANTED Waiting for input from rpotts
Updated•25 years ago
|
Whiteboard: HELP WANTED Waiting for input from rpotts → [NEED HELP] Waiting for input from rpotts
Whiteboard: [NEED HELP] Waiting for input from rpotts → [HELP WANTED] Waiting for input from rpotts
Updated•25 years ago
|
Assignee: radha → rpotts
Status: ASSIGNED → NEW
Comment 27•25 years ago
|
||
Rick, I'm reassigning this to you, so that it gets counted in your plate. Once
you have the loadgroup part working, you can re-assign back to me so that I
hook-up webshell to it.
Comment 28•25 years ago
|
||
"4) We are not clear about the privacy policy. ie., when it *should* be sent
irrespective of the user's preference and when it need not be sent. Can someone
comment on this?"
Referer need never be sent. Any servers relying on referrer being sent are
flawed.
<opinion>I would consider it a bug, and a security flaw, were Mozilla to
transmit referer despite my preference indicating that it should not, assuming
such a preference setting existed.</opinion>
Reporter | ||
Comment 29•25 years ago
|
||
Example of server relying on referrer: http://www.intellicast.com/
Most of the weather images within the site can only be viewed if the referrer is
the page that contains the image. It also has an ad. Otherwise people would
link straight to the images.
Comment 30•25 years ago
|
||
Sounds like another candidate for bug #7380 use.
Comment 31•25 years ago
|
||
*** Bug 16404 has been marked as a duplicate of this bug. ***
Comment 32•25 years ago
|
||
*** Bug 14129 has been marked as a duplicate of this bug. ***
Comment 33•25 years ago
|
||
*** Bug 14128 has been marked as a duplicate of this bug. ***
Updated•25 years ago
|
Summary: [NECKO] Send HTTP_REFERER value to server → [NECKO][webshell] [beta] Send HTTP_REFERER value to server
Target Milestone: M11 → M13
Comment 34•25 years ago
|
||
m13
Comment 35•25 years ago
|
||
For the paranoid, one may consider including a preference option to disable the
sending of the referrer, but no matter what this preference should either be
enabled or if there isn't one, the referrer should be sent.
Currently, referrer is the *only* way for webmasters to ensure that images or
other copyrighted files are viewed/downloaded exclusively from their own pages,
and not pirated by someone else.
Secondly, there's the issue of backwards compatibility: all previous Netscape
versions sent HTTP_REFERER. Not doing so in Mozilla will break many sites that
depend on this behavior (also exhibited by IE, Opera, Lynx, and others).
Finally, Mozilla's going to be rejected by webmasters if it doesn't send
HTTP_REFERER not only because their CGI scripts or other referrer-dependent
protections will break, but for the lack of alternative solution (that is at
least remotely reliaible).
Comment 36•25 years ago
|
||
*** Bug 19054 has been marked as a duplicate of this bug. ***
Comment 37•25 years ago
|
||
*** Bug 18331 has been marked as a duplicate of this bug. ***
Comment 38•25 years ago
|
||
*** Bug 21794 has been marked as a duplicate of this bug. ***
Comment 39•25 years ago
|
||
*** Bug 22131 has been marked as a duplicate of this bug. ***
Comment 40•25 years ago
|
||
*** Bug 20819 has been marked as a duplicate of this bug. ***
Comment 41•25 years ago
|
||
*** Bug 23319 has been marked as a duplicate of this bug. ***
Comment 42•25 years ago
|
||
*** Bug 19037 has been marked as a duplicate of this bug. ***
Updated•25 years ago
|
QA Contact: paulmac → tever
Updated•25 years ago
|
Target Milestone: M13 → M15
Comment 43•25 years ago
|
||
We definately need to deal with this all the way from the URI loader down
through the nsHTTPChannel... I'll look at this when I get back.
Comment 44•25 years ago
|
||
Not for alpha, but eventually, you MUST make sure you never let non-HTTP
referers into the real world. This is a security hole, because non-HTTP
protocols can have all sorts of secret stuff in them, like my FS layout (in
file://) or which news articles I read (in news:// or nntp://) or my passwords
(ftp:// in some cases).
Unlike HTTP URLs, these other protocols are NOT expecting to be exposed to
random web logs owned by potentially hostile people. So, Mozilla must prevent
them from escaping into REFERER at all. No useful web pages will be broken by
this if it's implemented correctly.
Comment 45•25 years ago
|
||
*** Bug 16587 has been marked as a duplicate of this bug. ***
Comment 46•25 years ago
|
||
I disagree with ruth@innocent.com. It can be very useful to see via the
referer when someone mentions my web site in a USENET posting. I would hate
to see Mozilla unilaterally decide to block this in all circumstances.
Updated•25 years ago
|
Keywords: helpwanted
Comment 47•25 years ago
|
||
*** Bug 25257 has been marked as a duplicate of this bug. ***
Comment 48•25 years ago
|
||
*** Bug 25877 has been marked as a duplicate of this bug. ***
Comment 49•25 years ago
|
||
*** Bug 25977 has been marked as a duplicate of this bug. ***
Comment 50•25 years ago
|
||
*** Bug 25063 has been marked as a duplicate of this bug. ***
Comment 51•25 years ago
|
||
*** Bug 26510 has been marked as a duplicate of this bug. ***
Comment 52•25 years ago
|
||
Due to Beta indication in Summary, putting beta1 into keyword field.
Keywords: beta1
Comment 53•25 years ago
|
||
*** Bug 26663 has been marked as a duplicate of this bug. ***
Updated•25 years ago
|
Summary: [NECKO][webshell] [beta] Send HTTP_REFERER value to server → [NECKO][webshell] Send HTTP_REFERER value to server
Whiteboard: [HELP WANTED] Waiting for input from rpotts → Waiting for input from rpotts
Comment 54•25 years ago
|
||
RickP: can we get your input on this. Is it essential for beta?
Reporter | ||
Comment 55•25 years ago
|
||
This prevents use of some major sites, for example, http://www.intellicast.com/
(which is probably the best source of US weather radar on the Net).
Not fixing this will also annoy webmasters because they won't get the referrer
field in their logs.
Comment 56•25 years ago
|
||
I don't disagree that it would be USEFUL to a web master to see which news
articles I read; he'd probably also find my PASSWORDS and CREDIT CARD NUMBER
useful, but that's no reason to create potential security problems. Bugtraq has
seen several "interesting" things sent as referer URLs, and restricting them to
HTTP-only seems like the most conservative solution.
Comment 57•25 years ago
|
||
*** Bug 16003 has been marked as a duplicate of this bug. ***
Comment 58•25 years ago
|
||
still need input from Rick Potts to determine PDT status.
Comment 60•25 years ago
|
||
Putting on PDT+ radar for beta1.
Whiteboard: Waiting for input from rpotts → [PDT+]Waiting for input from rpotts
Comment 61•25 years ago
|
||
this bug is really annoying. all sites on altern.org are viewable as text only
with mozilla...
example: http://www.altern.org/bmxzine/
an interesting test is to load this page with Nav4.x (it should display
correctly), then load one of the images in a new window and then reload the
image : due to the fact no referrer is transmitted during the reload, you get a
permission denied, as in mozilla.
i guess that the server puts this restriction in order not to be used as a mere
"images server"...
Comment 62•25 years ago
|
||
for the accuracy, i must add that this server blocks only .jpg files with the
referrer trick, not .gif (don't ask me why)
Comment 63•25 years ago
|
||
Herve: No need to complain, it's on PDT+ radar which means it should be fixed
very very soon. :-)
PDT: Product Development Team.
PDT+: A bug which MUST be fixed by beta1.
Comment 64•25 years ago
|
||
well, that's not really a complaint. just a way of submitting another "real life
example" of a server that won't give the images if no appropriate referrer is
transmitted.
Comment 65•25 years ago
|
||
Oh, right. In that case thanks!
Assignee | ||
Comment 66•25 years ago
|
||
fixed. waiting for review.
Here is what the "network.sendRefererHeader" pref means-
0- Won't send the "Referer" header in any case.
1- Will send it from links clicked on a document.
2- Will send it for image requests from a page as well as (1).
The default of this pref is 2.
Status: NEW → ASSIGNED
Whiteboard: [PDT+]Waiting for input from rpotts → [PDT+] ETA 02/22/00
Assignee | ||
Comment 67•25 years ago
|
||
fix checked in.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
Comment 68•25 years ago
|
||
verified referer header w/Trace32
NT 2000022408
Status: RESOLVED → VERIFIED
Comment 69•25 years ago
|
||
*** Bug 29264 has been marked as a duplicate of this bug. ***
Comment 70•25 years ago
|
||
*** Bug 34735 has been marked as a duplicate of this bug. ***
Comment 71•25 years ago
|
||
It looks like this bug may have reopened? See bug 34735 on M14 build 033112
Comment 72•25 years ago
|
||
Hey,
please tell me, if I am correct in this Bug.
Mozilla doesn't send a HTTP_REFERER when it loads a document
in an ILAYER.
Netscape and Microsoft send one.
I made a little testpage where I explain all details.
You can find it under: http://212.227.152.190/mozilla/
I hope I am correct in this bug. Else please tell me
the correct one.
Thanx,
Lars
Reporter | ||
Comment 73•25 years ago
|
||
lars@topsitez.de - Please file that as a separate bug report.
Comment 74•25 years ago
|
||
Reopening for MacOS platform.
Nominating for nsbeta2 since it crashes on MacOS9.
Comment 75•25 years ago
|
||
Crashes on:
- MacOS 2000-05-16-09-M16 Commercial
Works fine on:
- Win98, Linux 2000-05-16-09-M16 Commercial
Reporter | ||
Comment 77•25 years ago
|
||
Hmmm... what crashes? My test page? If so, does it have anything to do with
this bug?
(Also retitling to remove my silly Apache-ism.)
Summary: [NECKO][webshell] Send HTTP_REFERER value to server → [NECKO][webshell] Send HTTP Referer field to server
Assignee | ||
Comment 78•25 years ago
|
||
please file separate bugs for the new problem(s). thx. closing this again.
Status: REOPENED → RESOLVED
Closed: 25 years ago → 25 years ago
Resolution: --- → FIXED
Comment 79•25 years ago
|
||
Hmmnn....lemme try it again. I believe (and will confirm or deny this later)
that the page crashes when loading.n -kritzer
Comment 80•25 years ago
|
||
Okay, my bad. This bug does *not* cause a crash.
The bug still exists on the Mac. The section of the page that deals with
referer looks like this:
Referer: (none)
Which is incorrect according to this bug report. Lastest mac bits have been
flaky, so I'll check it one last time with today's (2000-05-18-xx-M16) bits.
Reporter | ||
Comment 81•25 years ago
|
||
When testing, make sure to click on the URL to get to the page, rather than type
it in (if you type in the url, (none) is correct).
Comment 82•25 years ago
|
||
Ahhhh...yes, well that's the trick, isn't it? <grin> Thanks for the clue,
dbaron.
Marking VERIFED FIXED on:
- MacOS9 2000-05-18-09-M16 Commercial Build
- Linux6 2000-05-18-09-M16 Commercial Build
- Win98 2000-05-18-09-M16 Commercial Build
Status: RESOLVED → VERIFIED
Comment 83•23 years ago
|
||
from: Michel Carroll <michel@swipe.plc.uk>
still not fixed in NN 6.1 AND Mozilla Build ID: 2001080110.
Note test bed: browser running on Win 2K sp2 client with Win 2K sp2 server
running ASP 3 on IIS5.
ASP O/P:
Request.ServerVariables("all_http"):
HTTP_ACCEPT:text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
HTTP_ACCEPT_LANGUAGE:en-us
HTTP_CONNECTION:keep-alive
HTTP_HOST:www.swipe.plc.uk
HTTP_USER_AGENT:Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.3)
Gecko/20010801
HTTP_CONTENT_LENGTH:33
HTTP_CONTENT_TYPE:application/x-www-form-urlencoded
HTTP_ACCEPT_ENCODING:gzip, deflate, compress;q=0.9
HTTP_ACCEPT_CHARSET:ISO-8859-1, utf-8;q=0.66, *;q=0.66
HTTP_KEEP_ALIVE:300
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Comment 84•23 years ago
|
||
Any particular conditions needed to reproduce? I just put together a simple
test case and it seems to work fine; got the following request (Linux build
2001090408):
GET /test.html HTTP/1.0
Host: lintra:1355
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3+) Gecko/20010904
Accept: text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
Accept-Language: en-us
Accept-Encoding: gzip, deflate, compress;q=0.9
Accept-Charset: ISO-8859-1, utf-8;q=0.66, *;q=0.66
Connection: close
Referer: http://www/~nephtes/test.html
Comment 85•23 years ago
|
||
I'm resolving and verifying this bug.
Look for existing Referer bugs and file new ones. Please file off 0.9.4 (coming
soon, NS 6.1 or the daily trunk build.
Keywords: testcase
Summary: [NECKO][webshell] Send HTTP Referer field to server → [NECKO][webshell] Send HTTP Referer field to server, "network.sendRefererHeader"
Comment 86•23 years ago
|
||
This bug still partially exists in build 2001090908.
When following a link through a simple click, the Referer-Header gets sent
(expected behavior). However when right-clicking and selecting "Open Link in New
Window", _no_ Referer gets sent (Which is definitely wrong). You can test this
on the Link posted by dbaron@fas.harvard.edu
But I don't think this bug really is a blocker. Major or even normal would be
more appropriate.
And as for the privacy issues: RFC 2068 chapter 14.37 has it's own collection of
SHOULD's and MUST NOT's on this topic. However, I feel that the discussion of
these issues are out of scope for this bug, and should rather be posted as an
enhancement query. Implementing a preference to specify whether Mozilla sends
cross-site Referers might be a good idea though.
Comment 87•23 years ago
|
||
hey gagan,
i don't know if i should mention this or not - but i'm actually looking at the
issue of passing the referer header along when new windows are targeted :-)
If that is *really* what this bug has digressed to, then maybe i should take it
over...
But don't stick me with *all* the 'referer' bugs :-)
-- rick
P.S.
It might be better to open a new bug specific to the window targeting situation!
Comment 88•23 years ago
|
||
Bug 48902. We stuck paul with that one.
I'm calling this one: RESOLVED/FIXED.
Please see my previous comment.
Status: REOPENED → RESOLVED
Closed: 25 years ago → 23 years ago
Resolution: --- → FIXED
Comment 89•23 years ago
|
||
re-VERIFIED, cause this has been working for ages (modulo the specific bugs
listed in the comments)
Status: RESOLVED → VERIFIED
Updated•20 years ago
|
Product: Core → Mozilla Application Suite
You need to log in
before you can comment on or make changes to this bug.
Description
•