Searching for messages...

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Searching for messages...

Michael Heydekamp
...still seems to be quite a strange thing in Roundcube.

Just did a search for the term "Ibiza" in the subject:

1) First of all, some messages even when searching in just one specific
   folder are not found at all. Although the subject clearly contains this
   string, a message with the subject
   "Auftragsbestätigung Düsseldorf-Valencia-Ibiza 12JUN16"
   is not being found. Looking at the source of the message, it becomes
   quite clear, why:

   > Subject:
=?UTF-8?Q?Auftragsbest=C3=A4tigung_D=C3=BCsseldorf-Valencia-Ibiz?=
   >  =?UTF-8?Q?a_12JUN16?=

   See...? The subject is folded right before the last character of "Ibiza",
   and the search routine apparently doesn't unfold such headers. Not good.

2) OTOH, a search for the same string "Ibiza" across ALL folders does find
   messages which do not contain this string in the subject at all and
   therefore shouldn't be found:

   "Re: Anfrage Mexiko-Miami 02DEC16 und Montréal-New York 10DEC16"
   "Re: IBZ - Vigo"
   "✈ Aktuelle Flugzeugverfügbarkeiten"
   "Re: EMPTY LEG C650 (CITATION III ) Availability"
   "Re: ZRH - DUS"

   Just a few examples of HUNDREDS of (wrong) matches. What's the search
   routine looking for? Apparently not only for "Ibiza", but for what else?

   Indeed, those messages DO contain the string "Ibiza" in the body, but
   I didn't ask for a search in the subject AND the body.

I still may have more issues with regards to searching messages which I did
report last year already, but that should be it for today.

Searching for messages is quite critical for me, as I'm using Roundcube not
(only) for private purposes - as you may see above.

Cheers,
--
Michael Heydekamp
Co-Admin freexp.de
Düsseldorf/Germany
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

A.L.E.C
On 06/08/2016 02:07 AM, Michael Heydekamp wrote:

>    Indeed, those messages DO contain the string "Ibiza" in the body, but
>    I didn't ask for a search in the subject AND the body.

These may be IMAP server issues. Enable imap_debug to see what is the
query and what is the result.

--
Aleksander 'A.L.E.C' Machniak
Kolab Groupware Developer        [http://kolab.org]
Roundcube Webmail Developer  [http://roundcube.net]
---------------------------------------------------
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

Brendan


On 16-06-07 10:27 PM, A.L.E.C wrote:
> On 06/08/2016 02:07 AM, Michael Heydekamp wrote:
>
>>    Indeed, those messages DO contain the string "Ibiza" in the body, but
>>    I didn't ask for a search in the subject AND the body.
>
> These may be IMAP server issues. Enable imap_debug to see what is the
> query and what is the result.

to elaborate on what alec said, roundcube uses the imap SEARCH command
to find messages - perhaps your imap server isn't searching in UTF7IMAP
encoded strings correctly (or at all).

i have a folder here with two messages (with subjects matching yours,
including a wrapped line for the subject):

Maildir/.myfolder/cur% zgrep -A1 Subject *
1465399407.P44335Q0M64750.smtpin01,S=34512,W=35061:2,:Subject: ibiza
party time
1465399407.P44335Q0M64750.smtpin01,S=34512,W=35061:2,:X-Spamscore: 0
1465399409.P45445Q0M30546.smtpin01,S=15699,W=15955:2,:Subject:
=?UTF-8?Q?Auftragsbest=C3=A4tigung_D=C3=BCsseldorf-Valencia-Ibiz?=
1465399409.P45445Q0M30546.smtpin01,S=15699,W=15955:2,: =?UTF-8?Q?a_12JUN16?=

and the imap server returns both of them in response to a search (with
either UTF-8 or ASCII):

a select myfolder
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 2 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1430938009] UIDs valid
* OK [UIDNEXT 3] Predicted next UID
* OK [HIGHESTMODSEQ 1] Highest
a OK [READ-WRITE] Select completed (0.003 secs).
a01 SEARCH CHARSET UTF-8 SUBJECT "ibiza"
* SEARCH 1 2
a01 OK Search completed (0.002 secs).
a02 SEARCH CHARSET US-ASCII SUBJECT "ibiza"
* SEARCH 1 2
a02 OK Search completed (0.000 secs).
a01 SEARCH CHARSET UTF-8 SUBJECT "Auftragsbest"
* SEARCH 2
a01 OK Search completed (0.000 secs).

if your imap server doesn't support searching in utf7imap encoded
strings, it'd pretty hard for roundcube to work around that, considering
that there are a huge number of ways that string could have been encoded
(depending on the charset it was encoded from). roundcube would need to
issue an imap search for every possibility, which isn't reasonable (and
would be incredibly slow, even if it were done).
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

A.L.E.C
On 06/08/2016 09:04 PM, Brendan wrote:
> to elaborate on what alec said, roundcube uses the imap SEARCH command
> to find messages - perhaps your imap server isn't searching in UTF7IMAP
> encoded strings correctly (or at all).

Brendan, thanks for the explanation, but just to be clear, UTF-7
encoding has nothing to do with searching. This encoding is used only
for mailbox names.

--
Aleksander 'A.L.E.C' Machniak
Kolab Groupware Developer        [http://kolab.org]
Roundcube Webmail Developer  [http://roundcube.net]
---------------------------------------------------
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

Michael Heydekamp
In reply to this post by A.L.E.C
Am 08.06.2016 07:27, schrieb A.L.E.C:
> On 06/08/2016 02:07 AM, Michael Heydekamp wrote:
>
>>    Indeed, those messages DO contain the string "Ibiza" in the body, but
>>    I didn't ask for a search in the subject AND the body.
>
> These may be IMAP server issues. Enable imap_debug to see what is the
> query and what is the result.

And I think the scope of Roundcube when searching across ALL folders is
simply not correct (on all folders being searched).

Bt I'll check what imap_debug will show.

And what about item 1) of my previous message (folded subject)? No response
to that one...?

Cheers,
--
Michael Heydekamp
Co-Admin freexp.de
Düsseldorf/Germany
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

Michael Heydekamp
In reply to this post by Brendan
Am 08.06.2016 21:04, schrieb Brendan:

> On 16-06-07 10:27 PM, A.L.E.C wrote:
>> On 06/08/2016 02:07 AM, Michael Heydekamp wrote:
>>
>>>    Indeed, those messages DO contain the string "Ibiza" in the body, but
>>>    I didn't ask for a search in the subject AND the body.
>>
>> These may be IMAP server issues. Enable imap_debug to see what is the
>> query and what is the result.
>
> to elaborate on what alec said, roundcube uses the imap SEARCH command
> to find messages - perhaps your imap server isn't searching in UTF7IMAP
> encoded strings correctly (or at all).

Dunno why you're bringing up an UTF7-issue? The subject we're talking about
was a) UTF8-encoded, and b) the encoding is not the issue at all. The issue
is the FOLDED subject.

> if your imap server doesn't support searching in utf7imap encoded
> strings, it'd pretty hard for roundcube to work around that, considering
> that there are a huge number of ways that string could have been encoded
> (depending on the charset it was encoded from). roundcube would need to
> issue an imap search for every possibility, which isn't reasonable (and
> would be incredibly slow, even if it were done).

Sorry, I don't get what you're trying to point at. I'm just saying that
Roundcube should unfold subjects (and other headers) before it checks if
they are matching a particular search. Currently it seems that it doesn't do
that.

Alec's response above is BTW not referring to this issue at all. He's
referring to item 2) of my initial message, which describes a totally
different problem.

Cheers,
--
Michael Heydekamp
Co-Admin freexp.de
Düsseldorf/Germany
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

Brendan


On 16-06-10 02:20 PM, Michael Heydekamp wrote:

> Am 08.06.2016 21:04, schrieb Brendan:
>> On 16-06-07 10:27 PM, A.L.E.C wrote:
>>> On 06/08/2016 02:07 AM, Michael Heydekamp wrote:
>>>
>>>>    Indeed, those messages DO contain the string "Ibiza" in the body, but
>>>>    I didn't ask for a search in the subject AND the body.
>>>
>>> These may be IMAP server issues. Enable imap_debug to see what is the
>>> query and what is the result.
>> to elaborate on what alec said, roundcube uses the imap SEARCH command
>> to find messages - perhaps your imap server isn't searching in UTF7IMAP
>> encoded strings correctly (or at all).
>
> Dunno why you're bringing up an UTF7-issue? The subject we're talking about
> was a) UTF8-encoded, and b) the encoding is not the issue at all. The issue
> is the FOLDED subject.

as alec pointed out, i was incorrect in mentioning UTF7IMAP, the problem
is with UTF8 (which is what i meant, i just typed the wrong encoding
method both times in the message).

>> if your imap server doesn't support searching in utf7imap encoded
>> strings, it'd pretty hard for roundcube to work around that, considering
>> that there are a huge number of ways that string could have been encoded
>> (depending on the charset it was encoded from). roundcube would need to
>> issue an imap search for every possibility, which isn't reasonable (and
>> would be incredibly slow, even if it were done).
>
> Sorry, I don't get what you're trying to point at. I'm just saying that
> Roundcube should unfold subjects (and other headers) before it checks if
> they are matching a particular search. Currently it seems that it doesn't do
> that.

roundcube calls your IMAP server for search (with those imap search
commands i had performed manually). roundcube isn't responsible for
unfolding the header during the search - your imap server is.

> Alec's response above is BTW not referring to this issue at all. He's
> referring to item 2) of my initial message, which describes a totally
> different problem.

oh, probably. the problem of it not matching that utf8 folded subject is
more interesting though. debug for the

going back to the folded subject one:

> Subject:
=?UTF-8?Q?Auftragsbest=C3=A4tigung_D=C3=BCsseldorf-Valencia-Ibiz?=
>  =?UTF-8?Q?a_12JUN16?=

roundcube asks the imap server to do a search for messages matching that
subject. if the imap server does not respond with this message in
response to that search, then your imap server is either not unfolding
the subject headers or not converting the subject into UTF8 before
trying to match against the subject.

if you telnet to the imap server and log in, you should be able to tell
which of the two it is:

a01 SEARCH CHARSET UTF-8 SUBJECT "est=C3=A4"
- if this returns the message, then the imap server is not converting
the subject to UTF-8 when trying to search (if it were, the =C3=A4 bit
of utf-8 encoding would not match)

a01 SEARCH CHARSET UTF-8 SUBJECT "Düsse"
- this also ought to return the message, but wouldn't if the imap server
was failing to convert the subject to UTF-8 while searching

a01 SEARCH CHARSET UTF-8 SUBJECT "12JUN16"
- if this does not return the message, then the imap server is not
folding the headers correctly when trying to search (since that 12JUN16
string is on the folder part of the header)

a01 SEARCH CHARSET UTF-8 SUBJECT "Ibiza"
- ought to return the message. but is apparently not (my imap server
does return both messages, and roundcube displays both as a result).

again, here's my imap server responding to those queries correctly
(message ID 2 has the same subject you reported above - the "a" utf8
encoded and the subject line wrapped, MID 1 is a message with Ibiza in
the subject, but not wrapped or utf8 encoded):

correctly finding the message:
a01 SEARCH CHARSET UTF-8 SUBJECT "ibiza"
* SEARCH 1 2
a01 OK Search completed (0.000 secs).

correctly not finding the utf8 encoding bits:
a01 SEARCH CHARSET UTF-8 SUBJECT "est=C3=A4"
* SEARCH
a01 OK Search completed (0.000 secs).

correctly finding the message with the wrapped part of the subject:
a01 SEARCH CHARSET UTF-8 SUBJECT "12JUN16"
* SEARCH 1
a01 OK Search completed (0.000 secs).

correctly finding a utf8 string in the subject part that isn't folded:
a01 SEARCH CHARSET UTF-8 SUBJECT "Düsse"
* SEARCH 1
a01 OK Search completed (0.000 secs).
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Searching for messages...

A.L.E.C
In reply to this post by Michael Heydekamp
On 06/10/2016 11:10 PM, Michael Heydekamp wrote:
> And what about item 1) of my previous message (folded subject)? No response
> to that one...?

Both issues are likely your IMAP server issues. Use debug to see what is
send by Roundcube to the server.

--
Aleksander 'A.L.E.C' Machniak
Kolab Groupware Developer        [http://kolab.org]
Roundcube Webmail Developer  [http://roundcube.net]
---------------------------------------------------
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
_______________________________________________
Roundcube Development discussion mailing list
[hidden email]
http://lists.roundcube.net/mailman/listinfo/dev