[Pyogp] Tasks

[Locklainn]

Here are some things I think we should start to answer:
1. What are the protocols we want to implement? The wiki page for the
Protocols has an (incomplete and not very defined) listing of some of
the protocols, but which of those do we want to implement?
(wiki.secondlife.com/wiki/Protocol) aka, what are some of the things we
want to be able to do with the Client Library?
2. Framework - it seems we have set out to design a framework for
testing. Are we writing a framework (like unit testing) or are we just
writing test suites that use the client library?

[Enus Linden]

Post by Locklainn
1. What are the protocols we want to implement? The wiki page for the
Protocols has an (incomplete and not very defined) listing of some of
the protocols, but which of those do we want to implement?
(wiki.secondlife.com/wiki/Protocol) aka, what are some of the things
we want to be able to do with the Client Library?

anything and everything, though, I do believe that we should prioritize.
Locklainn, might you be able to come up with a list of what you see out
there, and we can start from there? a rough sketch in my mind looks
something like:

1. login and establish/maintain presence
2. parse the data sent via the communication channels (event queue/UDP)
3. start building out functionality in the client

Post by Locklainn
2. Framework - it seems we have set out to design a framework for
testing. Are we writing a framework (like unit testing) or are we just
writing test suites that use the client library?

framework all the way. I'll help sketch that out once I get back in on
Monday.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 257 bytes
Desc: OpenPGP digital signature
Url : http://lists.secondlife.com/pipermail/pyogp/attachments/20080710/13bb2662/signature.pgp

[Christian Scholz]

Hi!

Post by Enus Linden

Post by Locklainn
1. What are the protocols we want to implement? The wiki page for the
Protocols has an (incomplete and not very defined) listing of some of
the protocols, but which of those do we want to implement?
(wiki.secondlife.com/wiki/Protocol) aka, what are some of the things
we want to be able to do with the Client Library?

anything and everything, though, I do believe that we should prioritize.
Locklainn, might you be able to come up with a list of what you see out
there, and we can start from there? a rough sketch in my mind looks
1. login and establish/maintain presence
2. parse the data sent via the communication channels (event queue/UDP)
3. start building out functionality in the client

I would like to soon have something which actually is usable from a
command line at least, like IM, checking friends online etc.

Post by Enus Linden

Post by Locklainn
2. Framework - it seems we have set out to design a framework for
testing. Are we writing a framework (like unit testing) or are we just
writing test suites that use the client library?

framework all the way. I'll help sketch that out once I get back in on
Monday.

I don't really get the difference as the unit test framework is indeed a
framework ;-)

Rough ideas here:

We need some way of configuring tests so that you can give
usernames/passwords, URLs etc. to the test so that they know what to
test. This can be maybe passed by via some .ini or XML-file.

We might want to have separate test suites for separate parts like agent
domain/region domain and also smaller modules (if some component is
known to not yet implement e.g. inventory then there is no need to test it).

We might need some way to setup certain testing scenarios. If the
inventory is empty we probably cannot retrieve something so we need to
make sure it's not empty before we test some GET (as an example).

All in all I think we can use the unit test framework and encapsulate it
in some command line tool which then can call the individual suites. A
way to pass configuration should also not be hard to be found. In Zope
we also use some feature called test layers where every layer can setup
some test environment. Some can then be bigger and more complete (but
take longer to initialize), some might be more lightweight which is
maybe enough to call lowlevel things which do not need so much setup.
Not sure we need that here though as it's probably more a networking
based test library anyway.

These are my rough thoughts on this.

-- Tao

[Locklainn]

Sounds good to me. I guess I agree with Tao on the framework issue. So
are we starting from scratch, creating a framework LIKE unit testing,
but not actually unit testing? Or are we just using unit as a base, and
adding functionality to it (pyUnit in this case)?

Infinity and I updated the wiki to include some objects to the test
harness and client library. Check them out and add anything else you can
think of. I included both Enus and Tao's notes from the last emails.

Post by Christian Scholz
Hi!

Post by Enus Linden

Post by Locklainn
1. What are the protocols we want to implement? The wiki page for
the Protocols has an (incomplete and not very defined) listing of
some of the protocols, but which of those do we want to implement?
(wiki.secondlife.com/wiki/Protocol) aka, what are some of the things
we want to be able to do with the Client Library?

anything and everything, though, I do believe that we should
prioritize. Locklainn, might you be able to come up with a list of
what you see out there, and we can start from there? a rough sketch
1. login and establish/maintain presence
2. parse the data sent via the communication channels (event queue/UDP)
3. start building out functionality in the client

I would like to soon have something which actually is usable from a
command line at least, like IM, checking friends online etc.

Post by Enus Linden

Post by Locklainn
2. Framework - it seems we have set out to design a framework for
testing. Are we writing a framework (like unit testing) or are we
just writing test suites that use the client library?

framework all the way. I'll help sketch that out once I get back in
on Monday.

I don't really get the difference as the unit test framework is indeed
a framework ;-)
We need some way of configuring tests so that you can give
usernames/passwords, URLs etc. to the test so that they know what to
test. This can be maybe passed by via some .ini or XML-file.
We might want to have separate test suites for separate parts like
agent domain/region domain and also smaller modules (if some component
is known to not yet implement e.g. inventory then there is no need to
test it).
We might need some way to setup certain testing scenarios. If the
inventory is empty we probably cannot retrieve something so we need to
make sure it's not empty before we test some GET (as an example).
All in all I think we can use the unit test framework and encapsulate
it in some command line tool which then can call the individual
suites. A way to pass configuration should also not be hard to be
found. In Zope we also use some feature called test layers where every
layer can setup some test environment. Some can then be bigger and
more complete (but take longer to initialize), some might be more
lightweight which is maybe enough to call lowlevel things which do not
need so much setup. Not sure we need that here though as it's probably
more a networking based test library anyway.
These are my rough thoughts on this.
-- Tao

[Christian Scholz]

Hi!

I just wanted to sketch out the next tasks I see for pyogp lib:

- Setup a test framework because right now we don't have any. I started
with some experimentation and will present this shortly.

- Refine components inside it. E.g. right now we convert an agent to
IPlaceAvatarAdapter, this maybe should not be the agent but the
agentdomain we adapt as it's the agentdomain which provides this call
and it shows better where the REST resource resides. An agent can still
be used to contain the name etc. (which we actually don't have right
now. A credential might contain it but that's not necessarily the case.
We might need some webservice to ask for the name and profile information).
Another thing I want to clean up a bit is serialization. Right now we
have a serializer for caps and one for the credentials. I would like
them to use the same interface as they both need to spit out LLSD. So my
idea would be to simply say

serializer = ISerialization(credentials) # or cap
data = serializer.serialize()

(we might also define __call__ instead of serialize. In this case it's
just serializer() to get the result. But this might be confusing)
All in all we only had one interface then for serializing something, not
one for credentials and one for caps and so on.

- discuss development process. When to create a branch, when not to. How
to communicate what you are going to change? I would like to keep lots
of communication going via this list (as it's more traceable) what we
are going to add/remove/change and also to maybe start creating branches
for some stuff. With svn it's unfortunately somewhat expensive and
annoying to do branching and merging (compared to hg or git) but I hope
we can still do this.

- Add the UDP loop. Here the question is how we do this best in library
form. Do we ask the application to create the loop and use functions of
the library to do the actual work or do we implement it in the library
with hooks for the application to hook in. And how do we do this without
depending on a certain networking library? And how do we do this to keep
it testable? This needs some thinking apparently. I would think that the
library would both contain a loop of it's own but it's optional to use
it. An application can also do it itself and just call the functions the
predefined loop would also call.

Somehow we also need to manage to keep this loop in a separate
thread/coroutine so that a shell like interface is possible.

- Implement the packet handlers for the loop which starts with
implementing parsers for these packets. I usually had no luck with
parsing them but Locklainn said, he knows how to do this. So this might
be a task then :)
Tests for this should be added as well of course where example packets
are parsed and maybe some with errors inside which hopefully get caught.
So some thinking is needed about possible error cases. Somebody could
actually already start with collecting such packets and thinking about
tests which tests all the features of them (different data types, blocks
etc.)

Once we have the loop running we can then think about TP which hopefully
gets more documented by then ;-) In the OpenSim patch I also noticed
that request_rez_avatar seems not to be used, just rez_avatar. But I
guess request_rez_avatar is not optional but more or less the seed cap
for the rez_avatar call.

In the OGP group we also need to discuss where the region domain comes
in (we did this once already) because right now you need to know the
region IP/name but you have no idea where to get that from. I think some
step inbetween which asks the region domain is necessary and maybe even
the regiondomain can then implement request_rez_avatar in some fashion
that it takes a regionname instead of an IP/DNS name. The name could
come from some mapping service which knows about all the regions.

I will also post the last bit to sldev later today.

Ok, enough mails.. work is waiting!
Please reply what you think about the tasks above.
And if you have questions about the ZCA please also reply to the
individual post of the tutorial. We can later also put this into the
wiki, for me writing it quickly as mail is just easier ;-)


-- Christian

[Timothy Loughlin (Locklainn Linden)]

I put Tao's "Tutorial" on the wiki:
https://wiki.secondlife.com/wiki/Pyogp/Client_Lib/Development/Zope

[Meadhbh S. Hamrick (Infinity Linden)]

lemme chime in with my $0.02

Post by Christian Scholz
We need some way of configuring tests so that you can give
usernames/passwords, URLs etc. to the test so that they know what to
test. This can be maybe passed by via some .ini or XML-file.

This is what we call "test fixtures." We can setup PyUnit to
initialize a number of "known good" test accounts prior to each test.
The details of avatars and regions can be hard coded in the setup
methods (easier) or read from a .ini or XML file (less easy, but
perhaps more flexible.)

Post by Christian Scholz
We might want to have separate test suites for separate parts like

agent

Post by Christian Scholz
domain/region domain and also smaller modules (if some component is
known to not yet implement e.g. inventory then there is no need to

test it).

Yup. And the purpose of the tests are threefold:

1. test code
1. a. test the PyOGP implementation
1. b. use PyOGP's library to test other implementations (like LL's and
OpenSim)

2. demonstrate proper use of the SLGOGP API.

3. communicate bugs or feature enhancements.

Post by Christian Scholz
We might need some way to setup certain testing scenarios. If the
inventory is empty we probably cannot retrieve something so we need

to

Post by Christian Scholz
make sure it's not empty before we test some GET (as an example).

yup. plus... there's a couple ways of thinking about this... a. you
could write unit tests that only test the pyogp lib code that handles
inventory and see that the code "does the right thing." b. you can
write a "live test" that actually goes over the network to an agent
domain and queries the inventory of an avatar we've explicitly setup
to have a null inventory. The former is good 'cause it's fast and
doesn't require you to coordinate with peeps who are managing an agent
domain. It also goes a long way to demonstrate confidence in the
codebase. The latter is good 'cause it's a more "system approach" and
will reveal non-code related problems (like we configured something
wrong on the LL side) or we're not compatible with a particular
revision of the agent domain server, etc.

Post by Christian Scholz
All in all I think we can use the unit test framework and

encapsulate it

Post by Christian Scholz
in some command line tool which then can call the individual

suites. A

Post by Christian Scholz
way to pass configuration should also not be hard to be found. In

Zope

Post by Christian Scholz
we also use some feature called test layers where every layer can

setup

Post by Christian Scholz
some test environment. Some can then be bigger and more complete (but
take longer to initialize), some might be more lightweight which is
maybe enough to call lowlevel things which do not need so much setup.
Not sure we need that here though as it's probably more a networking
based test library anyway.

I would love to see both a local test of the code and a "live" test,
for reasons given above. Definitely interested in a command line tool
that knows how to do either. If we wanted to get fancy, we could have
a graphical tool (like Squeak's SUnit Test Runner) that gives you
lotsa options for what tests to run.

-cheers
-Infinity

Oh... and P.S. I'm working on some of these tests and as soon as i
figure out what i can publish and when, i'll add them to the existing
code repository

Post by Christian Scholz
These are my rough thoughts on this.
-- Tao

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 486 bytes
Desc: This is a digitally signed message part
Url : http://lists.secondlife.com/pipermail/pyogp/attachments/20080711/fb23d2ce/PGP.pgp

[Locklainn]

Hey Tao,
How do I go about adding files to the codebase? and building? It seems I
have to run the who bootstrap, buildout process everytime, which is a
lot for a simple python script run.

[Christian Scholz]

Hi!

Post by Locklainn
Hey Tao,
How do I go about adding files to the codebase? and building? It seems I
have to run the who bootstrap, buildout process everytime, which is a
lot for a simple python script run.

If you add files to an existing package you don't need to run the
buildout command. It's only needed if you change the buildout.cfg.
bootstrap.py is only called once anyway, to create bin/buildout.

buildout also runs faster if called like this:

bin/buildout -No

-N : non-newest mode, don't install updated eggs if they are available
-o : work completely offline

adding -v or -vv can also help for debugging.

But as said, normally you don't need to run it.

As for adding files it depends where you want to add them btw. If it's
in pyogp.lib.base or any other existing package/egg then you can just
add them. The other thing would be creating a new package. This is
easiest done with a tool called paster which I wanted to describe on the
wiki. If you need it now, please tell me.

-- Tao

[Christian Scholz]

Hi!

Post by Meadhbh S. Hamrick (Infinity Linden)
lemme chime in with my $0.02

Post by Christian Scholz
We need some way of configuring tests so that you can give
usernames/passwords, URLs etc. to the test so that they know what to
test. This can be maybe passed by via some .ini or XML-file.

This is what we call "test fixtures." We can setup PyUnit to initialize
a number of "known good" test accounts prior to each test. The details
of avatars and regions can be hard coded in the setup methods (easier)
or read from a .ini or XML file (less easy, but perhaps more flexible.)

What I meant is more how you actually setup the remote part with a test
fixture as you might not have access to it directly. Some setup
mechanism seems to be needed here, so that you e.g. can tell OpenSim:
Load this test fixture.

Post by Meadhbh S. Hamrick (Infinity Linden)

Post by Christian Scholz
We might want to have separate test suites for separate parts like agent
domain/region domain and also smaller modules (if some component is
known to not yet implement e.g. inventory then there is no need to

test it).
1. test code
1. a. test the PyOGP implementation

I would do this not in the test harness but directly in the pyogp
library package (as I started to do), esp. when using test driven
development.

Post by Meadhbh S. Hamrick (Infinity Linden)
1. b. use PyOGP's library to test other implementations (like LL's and
OpenSim)
2. demonstrate proper use of the SLGOGP API.
3. communicate bugs or feature enhancements.

yup.

Post by Meadhbh S. Hamrick (Infinity Linden)

Post by Christian Scholz
We might need some way to setup certain testing scenarios. If the
inventory is empty we probably cannot retrieve something so we need to
make sure it's not empty before we test some GET (as an example).

yup. plus... there's a couple ways of thinking about this... a. you
could write unit tests that only test the pyogp lib code that handles
inventory and see that the code "does the right thing." b. you can write
a "live test" that actually goes over the network to an agent domain and
queries the inventory of an avatar we've explicitly setup to have a null
inventory. The former is good 'cause it's fast and doesn't require you
to coordinate with peeps who are managing an agent domain. It also goes
a long way to demonstrate confidence in the codebase. The latter is good
'cause it's a more "system approach" and will reveal non-code related
problems (like we configured something wrong on the LL side) or we're
not compatible with a particular revision of the agent domain server, etc.

I would see this basically as the difference in tests being in the
library package itself and those in test harness. The latter I suppose
to go through network and might be more live. Indirectly they of course
also test pyogp itself because we might also get test failures if we
send the wrong requests.

Post by Meadhbh S. Hamrick (Infinity Linden)

Post by Christian Scholz
All in all I think we can use the unit test framework and encapsulate it
in some command line tool which then can call the individual suites. A
way to pass configuration should also not be hard to be found. In Zope
we also use some feature called test layers where every layer can setup
some test environment. Some can then be bigger and more complete (but
take longer to initialize), some might be more lightweight which is
maybe enough to call lowlevel things which do not need so much setup.
Not sure we need that here though as it's probably more a networking
based test library anyway.

I would love to see both a local test of the code and a "live" test, for
reasons given above. Definitely interested in a command line tool that
knows how to do either. If we wanted to get fancy, we could have a
graphical tool (like Squeak's SUnit Test Runner) that gives you lotsa
options for what tests to run.

+1

Post by Meadhbh S. Hamrick (Infinity Linden)
Oh... and P.S. I'm working on some of these tests and as soon as i
figure out what i can publish and when, i'll add them to the existing
code repository

Great. I haven't had time to look at it yet but hopefully will do next
week then.

I personally also was discussing a way to do those not-live tests
without breaking the interface. My first test was with a mockup agent
domain server running which returns fixed values. The problem is though
that this needs to be started before running the tests. This needs to be
automated more and I have some idea on how. I will post about this more
the next days.

-- Christian