~samwhited/xmpp

1075c6b2d5343d51d0be7115820729fed86ccd46 — Sam Whited 1 year, 8 months ago 4123200
all: use xmlstream.Iter and remove internal/iter

The internal/iter package previously contained an API for iterating over
child elements and decoding them lazily.

This is needed by any package that exposes an iterator such as the
roster package, and is mentioned in the documentation
(https://mellium.im/docs/extensions) as the API to use when creating
your own extensions, but it was internal and not actually usable by this
package. Its own doc comment said that it would eventually be exported
when the API stabalized. It hasn't been necessary to change the API
since creating it so it became time to let it graduate to the
mellium.im/xmlstream module where it can be more broadly useful.

This patch bumps the version of xmlstream used and makes the minor
changes necessary to use the xmlstream version of the iterator. It also
removes the old internal/iter package and updates the documentation to
mention its new location as part of xmlstream.

They grow up so fast!

Signed-off-by: Sam Whited <sam@samwhited.com>
8 files changed, 16 insertions(+), 253 deletions(-)

M docs/extensions.md
M go.mod
M go.sum
D internal/iter/iter.go
D internal/iter/iter_test.go
M mux/mux.go
M receipts/receipts.go
M roster/roster.go
M docs/extensions.md => docs/extensions.md +9 -13
@@ 199,23 199,20 @@ if iter.Err() != nil {
```

Because iterators are common and all largely share the same logic to decode
child elements and return them, the [`internal/iter`] package was written to
make much of the logic reusable.
Because this package is internal you can't use it for your custom extensions
yet, but its types will be moved to an external package once we are sure that
the API is stable.
Instead of operating directly on decoded child elements, the `iter` package
operates on the token stream and returns access to each child element it finds,
letting your `Iter` type do the final decoding into a concrete type of your
choosing.
child elements and return them, much of the logic has been extracted into the
[`xmlstream.Iter`] type.
Instead of operating directly on decoded child elements, the `xmlstream.Iter`
type operates on the token stream and returns access to each top-level start
token it finds, letting your custom `Iter` type do the final decoding into a
concrete type of your choosing.
In the case of the `roster` package, this type is a `roster.Item`.

Most iterators will need to maintain some internal state.
This normally comprises any errors that were generated, a value representing the
last child element that was decoded, and an underlying `[`iter.Iter`].
last child element that was decoded, and an underlying `xmlstream.Iter`.

On your `Iter` type, most methods need to simply call the similarly named method
on the underlying `iter.Iter`.
on the underlying `xmlstream.Iter`.
The exceptions are the `Err` and `Next` methods.
The `Err` method should return any errors generated by your decoding first and
if no such errors exist return `iter.Err()`.


@@ 246,8 243,7 @@ func (i *Iter) Next() bool {
```


[`internal/iter`]: https://pkg.go.dev/mellium.im/xmpp/internal/iter
[`iter.Iter`]: https://pkg.go.dev/mellium.im/xmpp/internal/iter#Iter
[`xmlstream.Iter`]: https://pkg.go.dev/mellium.im/xmlstream#Iter
[`mellium.im/xmpp`]: https://pkg.go.dev/mellium.im/xmpp/mux
[`mellium.im/xmpp/mux`]: https://pkg.go.dev/mellium.im/xmpp/mux
[`mellium.im/xmpp/ping`]: https://pkg.go.dev/mellium.im/xmpp/ping

M go.mod => go.mod +1 -1
@@ 8,5 8,5 @@ require (
	golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
	golang.org/x/text v0.3.2
	mellium.im/sasl v0.2.1
	mellium.im/xmlstream v0.14.0
	mellium.im/xmlstream v0.15.0
)

M go.sum => go.sum +2 -2
@@ 17,5 17,5 @@ mellium.im/reader v0.1.0 h1:UUEMev16gdvaxxZC7fC08j7IzuDKh310nB6BlwnxTww=
mellium.im/reader v0.1.0/go.mod h1:F+X5HXpkIfJ9EE1zHQG9lM/hO946iYAmU7xjg5dsQHI=
mellium.im/sasl v0.2.1 h1:nspKSRg7/SyO0cRGY71OkfHab8tf9kCts6a6oTDut0w=
mellium.im/sasl v0.2.1/go.mod h1:ROaEDLQNuf9vjKqE1SrAfnsobm2YKXT1gnN1uDp1PjQ=
mellium.im/xmlstream v0.14.0 h1:vTljQmcFQq7LEb+LJQV0VI8wnuFnzBy1AnfUbA4SrL8=
mellium.im/xmlstream v0.14.0/go.mod h1:O7wqreSmFi1LOh4RiK7r2j4H4pYDgzo1qv5ZkYJZ7Ns=
mellium.im/xmlstream v0.15.0 h1:NczJZ5FYsRhaA2asw0/hrQm83K81cSTJszKhHh4s18Q=
mellium.im/xmlstream v0.15.0/go.mod h1:7SUlP7f2qnMczK+Cu/OFgqaIhldMolVjo8np7xG41D0=

D internal/iter/iter.go => internal/iter/iter.go +0 -117
@@ 1,117 0,0 @@
// Copyright 2019 The Mellium Contributors.
// Use of this source code is governed by the BSD 2-clause
// license that can be found in the LICENSE file.

// Package iter provides a streaming iterator over an XML elements children.
//
// This will likely be moved to mellium.im/xmlstream once the API is finalized.
package iter // import "mellium.im/xmpp/internal/iter"

import (
	"encoding/xml"
	"io"

	"mellium.im/xmlstream"
)

// Iter provides a mechanism for streaming the children of an XML element.
// Successive calls to the Next method will step through each child, returning
// its start element and a reader that is limited to the remainder of the child.
type Iter struct {
	r       xmlstream.TokenReadCloser
	err     error
	next    *xml.StartElement
	cur     xml.TokenReader
	closed  bool
	discard xmlstream.TokenWriter
}

type nopCloser struct{}

func (nopCloser) Close() error { return nil }

func wrapClose(r xml.TokenReader) xmlstream.TokenReadCloser {
	var c io.Closer
	var ok bool
	c, ok = r.(io.Closer)
	if !ok {
		c = nopCloser{}
	}

	return struct {
		xml.TokenReader
		io.Closer
	}{
		TokenReader: xmlstream.Inner(r),
		Closer:      c,
	}
}

// New returns a new iterator that iterates over the children of the most recent start
// element already consumed from r.
func New(r xml.TokenReader) *Iter {
	iter := &Iter{
		r:       wrapClose(r),
		discard: xmlstream.Discard(),
	}
	return iter
}

// Next returns true if there are more items to decode.
func (i *Iter) Next() bool {
	if i.err != nil || i.closed {
		return false
	}

	// Consume the previous element before moving on to the next.
	if i.cur != nil {
		_, i.err = xmlstream.Copy(i.discard, i.cur)
		if i.err != nil {
			return false
		}
	}

	i.next = nil
	t, err := i.r.Token()
	if err != nil {
		if err != io.EOF {
			i.err = err
		}
		return false
	}

	if start, ok := t.(xml.StartElement); ok {
		i.next = &start
		i.cur = xmlstream.MultiReader(xmlstream.Inner(i.r), xmlstream.Token(i.next.End()))
		return true
	}
	return false
}

// Current returns a reader over the most recent child.
func (i *Iter) Current() (*xml.StartElement, xml.TokenReader) {
	return i.next, i.cur
}

// Err returns the last error encountered by the iterator (if any).
func (i *Iter) Err() error {
	return i.err
}

// Close indicates that we are finished with the given iterator.
// Calling it multiple times has no effect.
//
// If the underlying TokenReader is also an io.Closer, Close calls the readers
// Close method.
func (i *Iter) Close() error {
	if i.closed {
		return nil
	}

	i.closed = true
	_, err := xmlstream.Copy(i.discard, i.r)
	if err != nil {
		return err
	}
	return i.r.Close()
}

D internal/iter/iter_test.go => internal/iter/iter_test.go +0 -113
@@ 1,113 0,0 @@
// Copyright 2019 The Mellium Contributors.
// Use of this source code is governed by the BSD 2-clause
// license that can be found in the LICENSE file.

package iter_test

import (
	"encoding/xml"
	"fmt"
	"io"
	"strconv"
	"strings"
	"testing"

	"mellium.im/xmlstream"
	"mellium.im/xmpp/internal/iter"
)

var (
	intStart = xml.StartElement{Name: xml.Name{Local: "int"}, Attr: []xml.Attr{}}
	fooStart = xml.StartElement{Name: xml.Name{Local: "foo"}, Attr: []xml.Attr{}}
)

var iterTests = [...]struct {
	in  string
	out [][]xml.Token
	err error
}{
	0: {in: `<a></a>`},
	1: {
		in: `<nums><int>1</int><foo/></nums>`,
		out: [][]xml.Token{
			{intStart, xml.CharData("1"), intStart.End()},
			{fooStart, fooStart.End()},
		},
	},
}

func TestIter(t *testing.T) {
	for i, tc := range iterTests {
		t.Run(strconv.Itoa(i), func(t *testing.T) {
			i := i
			_ = i
			d := xml.NewDecoder(strings.NewReader(tc.in))
			// Discard the opening tag.
			if _, err := d.Token(); err != nil {
				t.Fatalf("Error popping initial token: %q", err)
			}
			iter := iter.New(d)
			out := [][]xml.Token{}
			for iter.Next() {
				start, r := iter.Current()
				toks, err := xmlstream.ReadAll(r)
				if err != nil {
					t.Fatalf("Error reading tokens: %q", err)
				}
				if start != nil {
					toks = append([]xml.Token{start.Copy()}, toks...)
				}
				out = append(out, toks)
			}
			if err := iter.Err(); err != tc.err {
				t.Errorf("Wrong error: want=%q, got=%q", tc.err, err)
			}
			if err := iter.Close(); err != nil {
				t.Errorf("Error closing iter: %q", err)
			}

			// Check that the entire token stream was consumed and we didn't leave it
			// in a partially consumed state.
			if tok, err := d.Token(); err != io.EOF || tok != nil {
				t.Errorf("Expected token stream to be consumed, got token %+v, with err %q", tok, err)
			}

			// Don't try to compare nil and empty slice with DeepEqual
			if len(out) == 0 && len(tc.out) == 0 {
				return
			}

			if fmt.Sprintf("%#v", out) != fmt.Sprintf("%#v", tc.out) {
				t.Errorf("Wrong output:\nwant=\n%#v,\ngot=\n%#v", tc.out, out)
			}
		})
	}
}

type recordCloser struct {
	called bool
}

func (c *recordCloser) Close() error {
	c.called = true
	return nil
}

func TestIterClosesInner(t *testing.T) {
	recorder := &recordCloser{}
	rc := struct {
		xml.TokenReader
		io.Closer
	}{
		TokenReader: xml.NewDecoder(strings.NewReader(`<nums><int>1</int><foo/></nums>`)),
		Closer:      recorder,
	}
	iter := iter.New(rc)
	err := iter.Close()
	if err != nil {
		t.Errorf("Unexpected error: %v", err)
	}
	if !recorder.called {
		t.Errorf("Expected iter to close the inner reader if it is a TokenReadCloser")
	}
}

M mux/mux.go => mux/mux.go +1 -2
@@ 17,7 17,6 @@ import (

	"mellium.im/xmlstream"
	"mellium.im/xmpp"
	"mellium.im/xmpp/internal/iter"
	"mellium.im/xmpp/internal/ns"
	"mellium.im/xmpp/stanza"
)


@@ 413,7 412,7 @@ func forChildren(m *ServeMux, stanzaVal interface{}, t xmlstream.TokenReadEncode
	// TODO: figure out a good buffer size
	errs := make([]error, 0, 10)

	iterator := iter.New(r)
	iterator := xmlstream.NewIter(r)
	/* #nosec */
	defer iterator.Close()


M receipts/receipts.go => receipts/receipts.go +1 -2
@@ 14,7 14,6 @@ import (
	"mellium.im/xmlstream"
	"mellium.im/xmpp"
	"mellium.im/xmpp/internal/attr"
	"mellium.im/xmpp/internal/iter"
	"mellium.im/xmpp/internal/ns"
	"mellium.im/xmpp/mux"
	"mellium.im/xmpp/stanza"


@@ 63,7 62,7 @@ func (h *Handler) HandleMessage(msg stanza.Message, t xmlstream.TokenReadEncoder
		return err
	}

	i := iter.New(t)
	i := xmlstream.NewIter(t)
	/* #nosec */
	defer i.Close()


M roster/roster.go => roster/roster.go +2 -3
@@ 12,7 12,6 @@ import (

	"mellium.im/xmlstream"
	"mellium.im/xmpp"
	"mellium.im/xmpp/internal/iter"
	"mellium.im/xmpp/jid"
	"mellium.im/xmpp/mux"
	"mellium.im/xmpp/stanza"


@@ 25,7 24,7 @@ const (

// Iter is an iterator over roster items.
type Iter struct {
	iter    *iter.Iter
	iter    *xmlstream.Iter
	current Item
	err     error
}


@@ 130,7 129,7 @@ func FetchIQ(ctx context.Context, iq stanza.IQ, s *xmpp.Session) *Iter {

	// Return the iterator which will parse the rest of the payload incrementally.
	return &Iter{
		iter: iter.New(r),
		iter: xmlstream.NewIter(r),
	}
}