Wednesday, 11 April 2012< ^ >
arjan has set the subject to: Zotonic - the Erlang CMS
Room Configuration

[00:14:28] Maas-Maarten Zeeman leaves the room
[08:30:54] arjan joins the room
[08:48:26] arjan leaves the room
[09:03:08] arjan joins the room
[09:47:00] simon.smithies joins the room
[10:24:12] maas.maarten.zeeman joins the room
[11:22:52] <Marc Worrell> I think I need to add a field to the #context{}.
[11:23:32] <Marc Worrell> We need user-agent info for the template selection, also when there is no webmachine info in the context.
[11:24:17] <Marc Worrell> I also propose to add some extra info in the session: user-agent class, user-agent info, remote ip address
[11:28:38] <arjan> +1
[11:28:49] <maas.maarten.zeeman> +1
[11:28:58] <Marc Worrell> okidoki :)
[11:29:06] <arjan> :)
[11:29:31] <Marc Worrell> Then we can also make it possible (in due time) to see the sessions, and maybe hook into them for remote debugging or communication with the attached user.
[11:29:34] <maas.maarten.zeeman> Remote ip does also work with varnish :-)
[11:30:03] <Marc Worrell> You need to be careful with the remote-ip, it can be spoofed using the x-forwarded-for header.
[11:30:10] <maas.maarten.zeeman> yup.
[11:30:26] <Marc Worrell> we need to make that more robust, only allowing x-forwarded-for for known proxies.
[11:30:38] <Marc Worrell> till then, just don't trust it :)
[11:31:03] <Marc Worrell> when someone uses onion, will the remote-ip change between requests?
[11:32:47] <maas.maarten.zeeman> don't know
[11:35:00] <Marc Worrell> yep, every time a different ip address.
[11:35:11] <Marc Worrell> http://en.wikipedia.org/wiki/File:Tor-onion-network.png
[11:37:47] <maas.maarten.zeeman> Does tor act as a proxy or is it invisible?
[11:38:33] <maas.maarten.zeeman> I wonder how they handle cookies.
[11:38:53] <Marc Worrell> maybe we can use gproc for monitoring the sessions and pages
[11:38:59] <Marc Worrell> then we need less "managers"
[11:39:23] <Marc Worrell> tor forwards all requests through a network.
[11:39:36] <Marc Worrell> for the server it looks like the last tor node is the originating server
[11:40:02] <Marc Worrell> using DNS queries you are able to see if tor was used or not
[11:40:36] <Marc Worrell> (useful for sites that can be blocked or monitored, they can encourage using tor and give instructions how to do it)
[11:41:05] <Marc Worrell> ok, digging into the sessions/context etc for keeping the ua_class around
[11:41:28] <Marc Worrell> we might need APIs to render a template in the context of a particular session.
[11:43:22] <maas.maarten.zeeman> gproc also has a notification mechanism
[11:43:33] <Marc Worrell> yep - i just saw it :)
[11:43:56] <Marc Worrell> wondering if we can use it to replace some of our code.
[11:44:16] <Marc Worrell> our notifier is quite unique in that you can also register functions, not only processes.
[11:46:33] <Marc Worrell> will read later into it :) gproc might clean up quite a bit of code. less code is better :)
[11:47:25] <maas.maarten.zeeman> Later :-)
[11:47:48] <maas.maarten.zeeman> It is always possible to add to gproc
[11:47:50] <Marc Worrell> so little time, so much to do
[11:49:07] <maas.maarten.zeeman> They also have a global process registry. Wonder what will happen with a network split :-p
[11:52:33] <maas.maarten.zeeman> This also looks nice: https://github.com/uwiger/setup
[12:02:56] simon.smithies leaves the room
[12:17:13] <Marc Worrell> I will remove the session copying from the z_session_manager code - as that is one of the current bottle necks.
[12:17:27] <Marc Worrell> session copying -> context copying I mean
[14:55:02] <Marc Worrell> ok, remove context copying from the z_session_manager.
[14:55:34] <Marc Worrell> My simplistic apache bench benchmark went from ~400 pages/sec to 880.
[14:55:59] <Marc Worrell> correction: 920
[14:58:14] <Marc Worrell> correction: from ~700 -> ~920 req/sec.
[14:59:03] <Marc Worrell> it is about .28 msec/request less
[15:04:01] <maas.maarten.zeeman> Stop, otherwise elastic is obsolete! :-p
[15:07:03] <arjan> awesome
[15:08:14] <arjan> Marc -- are these changes cherry-pickable to be included in master?
[15:08:25] <arjan> was planning to do some 0.8 release work today :)
[15:08:54] <arjan> might be nice to mention a 30% performance increase in the change log :P
[15:09:31] <maas.maarten.zeeman> O yes.
[15:18:57] <Marc Worrell> removed one more context copy - now at 1100 req/sec
[15:19:27] <Marc Worrell> I think you can cherry pick them, didn't commit them yet.
[15:19:41] <Marc Worrell> There is still one crash with some session management, working on it.
[15:20:25] <Marc Worrell> hmmm, that 1100 was a quirk.
[15:21:11] <Marc Worrell> still at 920
[15:34:31] <arjan> anyone care for a release notes review?
[15:34:31] <arjan> https://gist.github.com/2359320
[15:34:52] <arjan> specially check if I've forgotten to mention something important..
[15:50:31] <Marc Worrell> Looks good, that was a lot of searching through the logs :)
[15:51:01] <Marc Worrell> Maybe also mention in the Core section that we stabilized the pgsql pool in the presence of database connection failures.
[15:51:27] <arjan> ok good one
[15:53:49] <Marc Worrell> I can now get 1919 req/sec on /lib/images/admin_zotonic.png
[15:54:05] <Marc Worrell> 3086 KB/sec
[15:54:09] <Marc Worrell> :)
[15:54:51] <Marc Worrell> without caching enabled on the lib resource.
[15:56:26] <Marc Worrell> and on a 57K image, using the image path: 1400/sec, 76500KB/sec
[16:01:02] <maas.maarten.zeeman> Never that is quite an improvement. Didn't expect it to be that much.
[16:04:19] <arjan> very nice :D
[16:11:35] <arjan> Marc, how is it going? release notes are finished...
[16:23:08] <maas.maarten.zeeman> Hey, i see i forgot to set read_concurrency to true for the observer table.
[16:23:55] <arjan> hmm
[16:24:23] <Marc Worrell> maybe we should also set read_concurrency for the templates table.
[16:24:30] <maas.maarten.zeeman> It is a R14 option.
[16:24:31] <Marc Worrell> = module_index table
[16:25:04] <Marc Worrell> it might be time to define the minimal Erlang system to R14? especially now that R15 is out.
[16:26:20] <arjan> eh yes
[16:26:28] <maas.maarten.zeeman> It is already R14 for the master right? There have been some incompatible renames (httpc)
[16:26:32] <arjan> we switched to httpc as well
[16:26:34] <arjan> indeed
[16:26:42] <arjan> R14 is the minimum
[16:26:49] <arjan> I'll update the docs
[16:26:58] <maas.maarten.zeeman> Ok. Nice for all the debian users out there.
[16:27:02] <maas.maarten.zeeman> NOT
[16:27:36] <maas.maarten.zeeman> Should we include erlang compile instructions?
[16:27:56] <Marc Worrell> the debian install is broken
[16:28:05] <Marc Worrell> maybe Basho has compile instructions?
[16:28:13] <maas.maarten.zeeman> so that they turn on ssl and other stuff zotonic needs
[16:28:14] <arjan> I always use a PPA
[16:28:15] <Marc Worrell> they have a R14 requirement
[16:28:18] <arjan> even on debian
[16:28:42] <arjan> I use this one: https://launchpad.net/~scattino/+archive/ppa
[16:28:44] <Marc Worrell> PPA? (apple boy looks fuzzy)
[16:29:07] <Marc Worrell> hmmm still need to update erlang on zotonic.com
[16:29:09] <arjan> Personal Package Archive
[16:29:41] <arjan> shall I say R14B01 or newer?
[16:30:12] <maas.maarten.zeeman> which of the R14 releases was broken? 2?
[16:31:12] <arjan> B04 I think
[16:31:16] <arjan> or 02, not sure
[16:31:37] <arjan> B02 it seems,
[16:31:38] <arjan> http://erlang.org/pipermail/erlang-questions/2011-June/059517.html
[16:31:59] <Marc Worrell> we have an open ticket for that broken release - as a reminder.
[16:33:11] <arjan> OK I'll say B03 or higher
[16:33:15] <maas.maarten.zeeman> Yeah, just google R14B02 broken :-)
[16:34:05] <Marc Worrell> :p
[16:34:27] <Marc Worrell> i should test R15 on my machine
[16:34:37] <Marc Worrell> then you get line numbers in the crashes
[16:34:52] <maas.maarten.zeeman> Still haven't tried that one either. Was hoping for some improvements in inets
[16:37:33] <Marc Worrell> btw, the dClass developer is picking up the problem report.
[16:37:40] <Marc Worrell> he found some small issues.
[16:38:07] <maas.maarten.zeeman> Nice
[16:38:27] <maas.maarten.zeeman> That was quite a debugging session last night.
[16:45:24] <arjan> Performance improvements - By tweaking the session manager we managed
to improved Zotonic's base throughput by a rough 30%.
klinkt goed?
[16:47:38] <maas.maarten.zeeman> Best wel. Dus 10x wordt 13x :-)
[16:48:24] <Marc Worrell> sounds great :)
[16:48:27] <Marc Worrell> and it is true
[16:49:30] <Marc Worrell> re: dClass - now we have to start juggling the ua_class state through the sessions and contexts, so that the correct templates get rendered using the push connections.
[16:50:06] <arjan> wow, indeed
[16:50:47] <Marc Worrell> it is fun :)
[16:53:16] <maas.maarten.zeeman> Just had something similar here. Rewireing
[16:53:43] <maas.maarten.zeeman> Wilt u efteling punten daarbij?
[17:02:04] <arjan> I do not really see performance increases for lib resources
[17:02:06] <arjan> using ab
[17:03:51] <arjan> 9592.52 [Kbytes/sec]
[17:04:00] <arjan> thats ~ 100Mbit
[17:04:32] <arjan> maybe my cpu's too fast and Im hitting an IO limit
[17:04:48] <maas.maarten.zeeman> Awesome
[17:05:00] <arjan> 3.3k req/sec
[17:05:41] <arjan> but also *without* marc's patch
[17:08:36] <maas.maarten.zeeman> Will check later on my main dev platform... an atom netbook.
[17:09:47] <arjan> :)
[17:10:01] <maas.maarten.zeeman> Maybve
[17:11:02] <maas.maarten.zeeman> the performance gain is more measurable then. :-)
[17:11:42] <arjan> yeah
[17:11:51] <arjan> at least the patch does not seem to hurt ;)
[17:12:06] <arjan> I'll include it in 0.8
[17:25:46] <maas.maarten.zeeman> When is the deadline for 0.8?
[17:28:03] <arjan> eh, right now
[17:28:14] <arjan> do you want to squeeze something in?
[17:28:38] <maas.maarten.zeeman> k :-) just wanted to check if adding read_concurrency actually works.
[17:29:08] <arjan> ok\
[17:29:09] <maas.maarten.zeeman> Well nobody missed it, and it is a lot faster anyhow :-)
[17:29:13] <arjan> hehe
[17:29:24] <arjan> just put it in master then
[17:30:10] <maas.maarten.zeeman> Was reading in the documentation that adding that option makes switching between reading and writing more expensive.
[17:30:49] <maas.maarten.zeeman> So good for the indexer and the observer. I turned turned it off for something in channel.
[17:31:36] <maas.maarten.zeeman> This machine doesn't have an easy master for zotonic :-( I'm jumping on my bike to home. Back in an hour or so.
[17:34:59] <maas.maarten.zeeman> Don't wait for me if you want to release earlier :-)
[17:36:11] maas.maarten.zeeman leaves the room
[18:30:11] arjan leaves the room
[18:42:06] maas.maarten.zeeman joins the room
[19:11:32] maas.maarten.zeeman leaves the room
[19:30:53] arjan joins the room
[20:18:59] <Marc Worrell> very nice release!
[20:19:15] <Marc Worrell> congratulations to the whole team :)
[20:20:31] <arjan> :D
[20:20:35] arjan leaves the room
[20:20:43] arjan joins the room
[20:20:50] <arjan> indeed
[20:22:15] <arjan> is new-admin-design ready voor unbranch denk je?
[20:22:35] <Marc Worrell> het is behoorlijk stabiel
[20:22:52] <Marc Worrell> en anderen kunnen nu van master naar 0.8 branch switchen
[20:23:25] <Marc Worrell> Ik ga even jouw release mail op erlang-questions zetten.
[20:29:52] <Marc Worrell> zag dat we zo'n 250 downloads per maand hebben - dat is heel netjes.
[20:30:02] <arjan> :)
[20:30:10] <arjan> oh ik dacht dat ik die lijst had gecc'd
[20:30:23] <Marc Worrell> zag het niet - en anders is het dubbel
[20:30:31] <arjan> apparently not
[20:30:32] <arjan> ok
[20:30:45] <Marc Worrell> (let's see what our non-dutch people will say when they see this discussion :)
[20:30:54] <arjan> indeed
[20:31:07] <Marc Worrell> They should learn Dutch :p
[20:31:39] <arjan> yes
[20:32:10] <arjan> contents of this chatroom is actually logged to disk
[20:32:18] <arjan> but not yet available online
[20:32:27] <Marc Worrell> nice history.
[20:32:41] <arjan> from oct. 2011
[20:33:40] <arjan> resource_static_files should be able to generate a directory index
[20:33:41] <arjan> :P
[20:33:49] <arjan> then it's easy to put online
[20:33:55] <Marc Worrell> yep
[20:34:18] <Marc Worrell> when we make those indices a bit nice then it might be very handy.
[20:34:23] <arjan> yes
[20:34:28] <arjan> templateable :)
[20:34:31] <Marc Worrell> shouldn't be too hard
[20:35:02] <arjan> indeed not
[20:35:05] <Marc Worrell> just one of those small things we never get to
[20:35:59] <Marc Worrell> de zotonic edoc moeten we updaten
[20:37:00] <arjan> heb ik al gedaan toch? Generated by EDoc, Apr 11 2012, 16:16:59.
[20:37:03] <Marc Worrell> is van 0.7 dev
[20:37:10] <arjan> alleen versienr. is vreemd idd
[20:37:19] <arjan> waar kwam dat ookalweer vandaan...
[20:37:39] <Marc Worrell> En de session spullen had je toch meegenomen? of niet?
[20:38:51] <arjan> jep
[20:38:52] <arjan> ah gevonden
[20:39:11] <Marc Worrell> de edoc van die session is ook oud
[20:40:02] <Marc Worrell> er is nu een z_session:set/2 maar die staat niet in de edoc
[20:40:05] <Marc Worrell> raar
[20:40:11] <Marc Worrell> server updaten?
[20:41:21] <arjan> fix ik zo
[20:41:22] <arjan> ogb
[20:56:58] Maas-Maarten Zeeman joins the room
[21:08:45] <Maas-Maarten Zeeman> Just tested new session manager on netbook. No difference here too. 95 req/sec, 500 kb/s
[21:09:54] <Marc Worrell> no diff?
[21:10:09] <Marc Worrell> i had a 30% diff on a 'hello world' page
[21:10:30] <Marc Worrell> cool: http://www.lemurproject.org/
[21:10:58] <Maas-Maarten Zeeman> Hmm, this is moderate complex couple of blocks. Nothing fancy
[21:14:06] <arjan> merged new-admin-design into master. let's see if my sites still work...
[21:14:33] <Marc Worrell> @maas - it was only about .3msec diff per req on my machine
[21:15:09] <Marc Worrell> the new-admin-design branch also brings the new module index
[21:15:29] <Marc Worrell> also felt a bit faster, though i didn't benchmark
[21:16:38] <Maas-Maarten Zeeman> That's the bottleneck then
[21:17:02] <Maas-Maarten Zeeman> When running zotonic etop -sort msg_q I see messages piling up.
[21:19:11] <arjan> lol, mediafonds site is severely broken on new master
[21:19:37] <arjan> actually I did not yet do much refactoring of zotonic's base templates on the frontend side... weird
[21:19:53] <Maas-Maarten Zeeman> @marc Or could erl on mac be different than erl on linux? Less copying maybe.
[21:20:13] <Marc Worrell> nope, must be copying between processes
[21:21:04] <Marc Worrell> where are the messages piling up?
[21:21:16] <Marc Worrell> @arjan, that is strange
[21:21:37] <arjan> yes, but I'll push it anyway, I've sent out a warning :)
[21:21:46] <Marc Worrell> :)
[21:21:55] <Marc Worrell> master = master = unstable
[21:22:28] <Maas-Maarten Zeeman> the module indexer.
[21:24:09] <arjan> pushing........
[21:24:29] <arjan> wow that takes long
[21:29:49] <Marc Worrell> quite some deltas
[21:30:10] <Marc Worrell> @maas staat je debug aan?
[21:30:20] <Marc Worrell> mod_developer
[21:30:59] <Marc Worrell> normally the module indexer shouldn't get many messages anymore.
[21:31:09] <Marc Worrell> only the 'all' variants go to the module indexer
[21:31:18] <Marc Worrell> are you sure you have the updated version?
[21:31:25] <Maas-Maarten Zeeman> on
[21:31:41] <Maas-Maarten Zeeman> :-)
[21:32:05] <Maas-Maarten Zeeman> k. normall I have it off. mod_develop is not nice on the atom.
[21:36:25] <Marc Worrell> when it is on then it is re-indexing… might be a bit harsh on the little machine
[21:36:34] <Maas-Maarten Zeeman> :-)
[21:36:39] <Marc Worrell> (we must find a better way, maybe inotify?)
[21:38:08] <Maas-Maarten Zeeman> running without mod_develop is fine. You just have to manually flush every now and then
[21:38:59] <Marc Worrell> z:m() does that already
[21:39:02] <Maas-Maarten Zeeman> No diff also :-) strange
[21:39:08] <Marc Worrell> weird
[21:39:12] <Maas-Maarten Zeeman> without mod_develop running
[21:39:25] <Marc Worrell> i really had ~400/sec a week ago and now ~900/sec
[21:39:49] <Marc Worrell> then there must be another bottle neck on the little one
[21:39:55] <Maas-Maarten Zeeman> Now checking out the new admin branch see what that gives
[21:40:11] <Marc Worrell> take the master - it is being merged :p
[21:40:17] <Maas-Maarten Zeeman> k
[21:40:22] <Marc Worrell> *living on the edge*
[21:40:33] <Marc Worrell> Arjan is doing mega-pushes
[21:41:48] <Marc Worrell> not yet pushed
[21:42:05] <Maas-Maarten Zeeman> k
[21:42:39] <Maas-Maarten Zeeman> making new-admin -desing then :-)
[21:42:41] <Marc Worrell> aha, try new-admin-design, that one has the new module indexer
[21:42:56] <Marc Worrell> might help on your little 'book
[21:43:12] <Maas-Maarten Zeeman> My precious :-)
[21:44:38] <Maas-Maarten Zeeman> building takes a while
[21:46:18] maas.maarten.zeeman joins the room
[21:46:51] maas.maarten.zeeman leaves the room
[21:50:04] <Maas-Maarten Zeeman> Hmm, doesn't start because of race condition in mod_signal. Fixed that in the master
[21:50:31] <Marc Worrell> then wait for the merge :p
[21:50:46] Maas-Maarten Zeeman leaves the room
[21:51:20] Maas-Maarten Zeeman joins the room
[21:52:06] Maas-Maarten Zeeman leaves the room
[21:52:09] maas.maarten.zeeman joins the room
[21:52:27] <maas.maarten.zeeman> heh.
[21:52:44] <maas.maarten.zeeman> slots got called before the module was started.
[21:53:12] <maas.maarten.zeeman> just send back [] then
[21:53:34] <maas.maarten.zeeman> back to db pool then
[21:53:52] <maas.maarten.zeeman> Can now share connection to memory db :-)
[22:03:58] <Marc Worrell> spending some time with Miss L :) *wave*
[22:04:30] <maas.maarten.zeeman> bye, good evening
[23:09:06] arjan leaves the room
Powered by ejabberd Powered by Erlang Valid XHTML 1.0 Transitional Valid CSS!