Thursday, 5 September 2013< ^ >
Arjan has set the subject to: Zotonic - The Erlang Web Framework & CMS
Room Configuration

[00:21:29] cillian.deroiste leaves the room
[00:48:26] <Marc Worrell> how are the docs on zotonic.com arranged? seems that the "latest" is referring to 0.9 instead of 0.10dev
[08:43:32] Maas joins the room
[08:48:57] arjan joins the room
[08:59:39] <Kaos> Marc Worrell: I think the docs build is failing for master...
[08:59:46] <Kaos> check with arjan..
[08:59:56] <arjan> I hear my name
[09:00:01] <Kaos> ;)
[09:00:26] <arjan> we need to tie building the docs into travis ci :)
[09:02:56] <arjan> for some reason make stubs takes ages
[09:03:14] <Kaos> oh.. do they now?
[09:03:24] <Kaos> maybe time to look things over at that end..
[09:03:29] <arjan> heh
[09:03:37] <arjan> its not really very verbose
[09:03:45] <Kaos> indeed not..
[09:04:01] <Kaos> I do recall it was a tad slow..
[09:04:39] <Kaos> but that can't be the reason why the 0.10 docs aren't built, right?
[09:04:55] <arjan> make stubs seems to hang here
[09:05:08] <Kaos> that sounds more like a reason, yeah.. :p
[09:05:24] <Kaos> I'll look into building the docs on master..
[09:05:33] <Kaos> see if have the same issue here..
[09:05:45] <arjan> not on zotonic.com
[09:05:49] <arjan> cool
[09:05:50] <Kaos> btw, are you sure you get a clean checkout?
[09:06:09] <Kaos> when switching from release-0.9.x to master..
[09:12:44] <arjan> I dont know really
[09:12:52] <arjan> havent looked in to it lately
[09:12:59] <arjan> probably its not clean
[09:13:10] <Marc Worrell> i think the deps keep "hanging around"
[09:13:23] <Marc Worrell> and all your local-no-git files stay
[09:13:51] <Kaos> ah, perhaps run a make clean, which will have to know how to get rid of such things..
[09:14:25] <Marc Worrell> yep - is there als a very-clean? like in removing the downloaded deps?
[09:14:33] <Marc Worrell> would be nice to test clean builds
[09:14:42] <Kaos> but still, the .generate scripts wouldn't heart from a little more love..
[09:29:49] <Kaos> why, it seems like a page session ends due to inactivity even tough it has a page open on the client end with a connected websockets stream.. so when I come back after the page session has ended, all I got over the stream is 204 No Content.. (it's my hunch at the moment, will investigate)..
[09:34:28] <Marc Worrell> strange, locally I have connections (with localhost) open over the night.
[09:34:56] <Marc Worrell> No activity over those - though I think we should do a periodic ping/pong over that line.
[09:35:56] <Kaos> hmm... looking at controller_postback:forbidden/2 there's a todo note about not making a new session.. so it does call ensure_all there...
[09:36:06] <Kaos> which ought to make sure there is a session, right?
[09:36:36] <Kaos> this is over localhost, it's been open overnight, and it doesn't work (I've seen this over lunch as well)
[09:36:46] <Kaos> I'm pretty sure it will work once I refresh the page..
[09:37:22] <Kaos> looking at a trace dump for the request now... some wading due to all context args filling the log..
[09:38:11] <Kaos> I know what I'd love to have, a trace log reader which can fold erl terms in the log... :)
[09:38:48] <Maas> Indeed, those are horrific.
[09:39:41] <Kaos> (in case you're wondering what trace I'm talking about, I simply did: dbg:tracer(). dbg:p(all, call). dbg:tpl(controller_template, []). and then made a request, voila ;)
[09:40:53] <Maas> btw. i'm refactoring z_lib_include a bit so it uses the dispatcher to create a url.
[09:49:41] arjan leaves the room
[09:55:44] <Kaos> aha! I call z_context:add_script_page/2 .. guess it would work better to simply add the script to the context directly and return that..
[10:36:28] <Kaos> uhu, that didn't solve it, though..
[10:36:50] <Kaos> why isn't the #context.render part also included in the data collected by z_script:get_script?
[10:37:03] <Kaos> ping Marc Worrell
[10:37:11] <Maas> o no s
[10:37:43] <Kaos> Maas?
[10:37:54] <Maas> I think you need to call split_script first
[10:38:07] <Kaos> huh? ok, will check
[10:38:37] <Maas> It fetches the script from the context, then you can use add_script.
[10:38:45] <Maas> depends a bit on what you do.
[10:39:13] <Kaos> I want to render a template and get the result back as response data to a postback request..
[10:39:23] <Kaos> but in this case, the page session is dead..
[10:39:35] <Maas> o, that doesn't help
[10:39:37] <Kaos> perhaps a better approach is to make sure the page session is brought back to life.. ?
[10:39:59] <Maas> why is it gone? No stream?
[10:40:18] <Kaos> no, the stream is there, but I suppose the page session has timed out and gone away..
[10:40:20] <Maas> Then it will travel to the client with the next postback
[10:40:46] <Maas> If there is a ws or comet attached it should not timeout.
[10:40:49] <Kaos> my postback context has no sessions in them
[10:41:27] <Kaos> well, they do once I reload the page, but that's the point, I don't want to reload.. it should just work..
[10:42:01] <Kaos> guess I will have to look into the z_context:ensure_all...
[10:42:02] <Maas> and you have a {% stream %} tag on the page?
[10:42:16] <Kaos> yes yes.. otherwise it wouldn't work at all, right?
[10:42:34] <Kaos> and it does work most of the time, it's just when it's been idle for some time it stops..
[10:42:42] <Kaos> the postbacks go fine, but I don't get the data back
[10:42:47] <Kaos> I get 204 no content
[10:42:58] <Kaos> all though it did render the template and all
[10:42:59] <Maas> No indeed, that should get you a comet or ws pid in your page session and it should keep it from timing out
[10:43:25] <Kaos> so, if it didn't time out, it died, or failed to pick it up (hence I'll look into ensure_all)
[10:43:52] <Kaos> and I got my cookies in the request too...
[10:43:54] <Maas> strange.
[10:44:16] <Kaos> and this is on localhost, no sleep, so no expected network issues..
[10:44:30] <Kaos> yeah..
[10:44:34] <Maas> try another browser maybe. chrome sometimes has problems with localhost connections
[10:45:01] <Maas> especially on non-standard ports. something is goofy in the omnibar..
[10:45:23] <Kaos> uh... do you mean that the ws connection may have gone down, let the page timeout, then reconnect the ws later and then the page session is by long gone..
[10:46:01] <Kaos> shouldn't z_context:ensure_all recreate the page session for me?
[10:46:06] <Maas> It could be that chrome is trying to setup a ws... sometimes that takes ages... it could easily be that the page session times out by then
[10:46:13] <Marc Worrell> we need page-process restarts — if no page process, start a new one and tell the html about it
[10:46:41] <Maas> There is a timeout when setting up the stream. If it takes too long to setup a ws it should switch to comet.
[10:47:00] <Maas> maybe the timing is a bit tight.
[10:47:01] <Kaos> but but but... the ws seems fine, it's the page session that's missing
[10:47:09] <Maas> Do you have a ws?
[10:47:13] <Kaos> oh..
[10:47:25] <Kaos> is there a easy way to tell?
[10:47:36] <Kaos> ah, right..
[10:47:48] <Maas> Sometimes it stays on "pending"... then it actually didn't send anything to the server.
[10:47:49] <Kaos> does this answer it? X-Requested-With:XMLHttpRequest
[10:48:04] <Kaos> well, I do get the roundtrip just fine, request response.
[10:48:40] <Maas> that sounds like comet
[10:49:16] <Kaos> aahh... and comet is one shots like http, right?
[10:49:45] <Marc Worrell> should have more retries - this really needs to become more stable
[10:49:47] <Kaos> so no open stream connection with the server.. ? (sorry, not really up-to-speed when it comes to how comet/ws works on the wire)
[10:50:06] <Kaos> Marc Worrell: I don't need retries here, the request goes just fine
[10:50:12] <Kaos> I just miss my page session
[10:50:21] <Maas> For comet there are retries, for ws not.
[10:50:46] <Marc Worrell> Poor @kaos, you must feel lonely without your page session
[10:50:51] <Kaos> I can even live without my page session for this, as long as the response data is sent back
[10:50:58] Kaos :p
[10:51:18] <Maas> blah.
[10:51:25] <Kaos> and it will be, as soon as I get z_script:get_script to fetch it
[10:51:52] <Maas> Try firefox, or safari.
[10:51:55] <Maas> :p
[10:52:12] <Marc Worrell> maybe add some lager debug stuff in the z_session_page — better to see what is happening in that quite essential piece of code
[10:52:34] <Kaos> I can see exactly what is happening, what do you want to know?
[10:52:55] <Kaos> ah, sorry.
[10:53:06] <Kaos> I read that a bit too fast, perhaps..
[10:53:23] <Maas> What I saw happening a while ago is that sometimes the js in chrome tries to setup a ws, but it doens't send out any data (they are forgetting to flush the socket I think)
[10:53:28] <Kaos> what I meant is that I can add traces dynamically, don't need to pester the code with debug prints..
[10:54:31] <Maas> Then after new WebSocket... it just sits there, no incoming data, no nothing.
[10:54:40] <Kaos> ok, I see two issues, 1) the page session went missing, and wont come back; and 2) I want to render a template and get the result back as response data to a postback request
[10:55:25] <Kaos> 2 I can fix quite easily.. either just find the right way to call on the template or fix it up so the rendered result ends up in a sensible place
[10:55:52] <Kaos> 1 needs some debugging to figure out why it went missing, or why it want create a new one..
[10:56:23] <Kaos> perhaps we don't want page sessions staying around for too long that's been idle, if we can just recreate them when we do get some action for them..
[10:56:49] <Maas> There is a timeout after two seconds (from the top of my head)... if there is no response from the websockets it tries to setup comet... but maybe that timeout is a bit too long or the page_session and it is already gone.
[11:03:18] <Maas> Tricky business... setting up these streams
[11:03:38] <Maas> Especially on mobile devices...
[11:05:58] <Maas> A lot of providers have proxies on port 80.. these block ws.. sometimes they also prevent long polling comet...
[11:10:01] <Marc Worrell> the 0.10dev docs are back - some local files were blocking the git co
[11:10:54] <Kaos> ah.. the docs should really do git co --force to get passed that
[11:10:54] <Marc Worrell> maybe we can also make a transport over port 1883 (MQTT :) )
[11:12:25] <Maas> We found that using port 443 is the most reliable... but still not fool proof.
[11:13:07] <Marc Worrell> maybe we need a state diagram, for making reliable connections, and retrying on different levels.
[11:13:29] <Maas> Yes big time.
[11:13:35] <Maas> Gonna be a big one.
[11:13:45] <Marc Worrell> would also be nice if a websocket connection sees that we need a new session, reports back to the page to retry with http, to set a session cookie, and then we can retry
[11:13:54] <Marc Worrell> state machine :p
[11:14:23] <Marc Worrell> use Graphviz
[11:14:25] <Kaos> any objections to also getting z_context:output when collecting response data in controller_postback?
[11:14:48] <Marc Worrell> no idea :p
[11:14:53] <Maas> Also nice that you can't call ws.close() if it wasn't open.
[11:14:59] <Kaos> heh, I'll go ahead and try it then.. ;)
[11:15:03] <Maas> It can stay in pending state.
[11:15:27] <Marc Worrell> arggghhhhh - we need a NFA for those connections.
[11:15:29] <Maas> and then suddenly after 2 minutes chrome sends the handshake
[11:15:40] <Maas> but then you already have a comet connection...
[11:15:44] <Marc Worrell> Even bigger DFA then
[11:16:50] <Maas> As setup ws connection via port 443 as stop gap measure.
[11:16:57] <Marc Worrell> start comet & ws with timers, if ws comes back, cancel comet timer or comet connection (hmmm, prob can't cancel the comet connection...)
[11:17:20] <Maas> It is horrific.
[11:18:20] <Maas> Maybe it is better to look for a reliable js lib which has such a DFA and the we just add some zotonic handlers for it.
[11:18:32] <Marc Worrell> it is a great state machine! more complicated than the machine of Charlie and the Chocolate Factory
[11:19:38] <Marc Worrell> yes - a lib is better - then we can also start with Server-Sent-Events
[11:19:55] <Maas> A lot of bigger sites with push still use old fashioned jsonp
[11:21:14] <Marc Worrell> maybe start contributing to http://socket.io ?
[11:21:56] <Marc Worrell> and maybe we should replace our cross-domain iframes with jsonp - much easier than that iframe business
[11:22:00] <Kaos> yay, I got my response data! (still no sessions: controller_postback,undefined,undefined,undefined,undefined, )
[11:24:07] <Maas> @marc pff, yes. You also have sock.js
[11:24:08] <Marc Worrell> poor @kaos - still so lonely and denied by the server
[11:24:28] <Marc Worrell> socket.io seems to be used by the node.js boys and girls
[11:24:40] <Marc Worrell> so every hip feature possible goes into there :p
[11:25:55] <Maas> haha... indeed, sock.js seems to be for the rest. I see erlang (cowboy) implementation for that.
[11:27:00] <Kaos> wait a minute, does comet work even if I don't have a {stream} tag on the page?
[11:27:06] <Marc Worrell> I'll make an issue for this socket stuff, then we can discuss and decide on a library here
[11:27:15] <Marc Worrell> @kaos… uhhh no
[11:27:23] <Marc Worrell> then it is not started
[11:27:56] <Marc Worrell> I have been wondering about turning that upside down, add a "nopush" option to {% script %}
[11:28:00] <Kaos> so why are my postbacks sent using comet, when on the admin they are sent over ws.. ?
[11:28:24] <Marc Worrell> in the admin there is a {% stream %}
[11:28:54] <Marc Worrell> and the postbacks you see arrive on the postback_controller, which also will try to pick up any data lingering around at the page process
[11:29:42] <Kaos> aha, I added a {% stream %} tag, and now it uses ws..
[11:29:59] <Kaos> so that is why my page session went missing in the first place
[11:30:20] <Kaos> but I can then confirm that comet works without a stream tag..
[11:32:50] <Marc Worrell> https://github.com/zotonic/zotonic/issues/646
[11:33:04] <Kaos> adding z_context:output to controller_postback was no good idea... caused validations to be added twice, for one thing..
[11:33:12] <Marc Worrell> uh oh :p
[11:34:29] <Kaos> reverting.. (never committed)
[11:34:53] <Kaos> but now I don't expect my page session to disappear, so it's a non issue, really..
[11:35:04] <Kaos> but I got another idea in the mean time..
[11:35:35] <Kaos> how about adding a #template{} update to the context.update list for rendering templates in the get_scripts function?
[11:35:47] <Maas> Thanks marc. I'll add my thoughts and observed problems to the issue. I've also recently rolled my own connection js library... :p
[11:36:01] <Kaos> (i.e. for queueing a template to be rendered and get the output as script along with all other actions )
[11:36:12] Maas leaves the room
[11:42:13] <Marc Worrell> Updated the issue with two libs https://github.com/zotonic/zotonic/issues/646
[11:43:02] <Marc Worrell> @kaos where should that output of that #template{} then go?
[11:43:42] <Marc Worrell> Documentation! http://zotonic.com/docs/latest/ref/modules/mod_mqtt.html
[11:43:52] <Kaos> +10 :)
[11:44:31] <Kaos> Marc Worrell: let me get back to you on that, when I have a proof of concept..
[11:48:48] arjan joins the room
[12:07:33] Maas joins the room
[12:15:51] <Kaos> Marc Worrell: ok, I'm ready to get back to you now.. : https://github.com/zotonic/zotonic/commit/b36811313ed10031ce09837e2c86dd86f58bd8fd ;)
[12:16:25] <Kaos> it's like two lines of code.. :p
[12:19:21] arjan leaves the room
[12:23:23] <Kaos> I came to my senses and implemented it as an action ;)
[12:28:47] <Marc Worrell> :-) easy way first
[12:33:11] <Kaos> yeah, I like how little was needed to get what I was after ;)
[12:40:03] arjan joins the room
[12:58:20] Maas leaves the room
[13:06:46] Maas joins the room
[13:07:39] Maas leaves the room
[13:09:27] Maas joins the room
[13:24:14] <arjan> Maas: thanks for the advice on mobile providers and websockets :)
[13:24:29] <arjan> good they leave https alone :)
[13:24:56] <arjan> Im too lazy to implement a fallback :p
[13:25:13] <arjan> its for the silent mobile disco this saturday, https://silentmobiledisco.nl
[13:33:34] <Maas> There is actually an obscure config for that.
[13:37:15] arjan leaves the room
[13:37:34] arjan joins the room
[13:47:08] <Maas> {websockethost, "silentmobiledisco,nl:443"},
[13:47:48] <Maas> Opens normal unencrypted websocket over port 443... you must direct your fw to the normal zotonic port then.
[13:52:31] <arjan> I just serve the whole site on HTTPS
[13:52:32] <arjan> :)
[13:52:48] <Maas> easy does it.
[13:52:58] <Maas> Behind a proxy?
[13:59:56] <arjan> on
[13:59:58] <arjan> *no
[14:00:35] <Maas> Actually I'm busy fixing some https related issues...
[14:16:25] <Maas> If you use m.site.hostname you will get the http hostname + port not ssl.
[14:16:50] <Maas> and if you use use_absolute_url for scripts and css you have the same issue.
[14:17:16] <Maas> Edge cases, everything else works fine.
[14:32:48] <Maas> Sometimes I get this with mod_ssl when I flush the site...
[14:32:51] <Maas> > z:flush
(zotonic001@zotonic-dev)2> ().
(zotonic001@zotonic-dev)3> 12:31:59.733 [error] z_notifier:404 Error folding {mod_ssl,pid_observe_dispatch_rules,<0.339.0>} with event dispatch_rules. Detaching pid.
12:31:59.760 [error] gen_server z_sites_dispatcher terminated with reason: no function clause matching z_sites_dispatcher:normalize_streamhosts([{error,{notify_observer,<0.339.0>,dispatch_rules,error,{case_clause,keep}}}], [{wm_host_dispatch_list,zotonic_status,undefined,undefined,undefined,[],true,[{home,[],controller_zotonic_status,...},...]}])
[14:33:19] <Maas> It doesn't recover from that.
[14:33:37] <Maas> Need to stop and start the site
[14:41:26] <Maas> Oops... typo in config.
[14:41:44] <Maas> nice one...
[15:04:47] Maas leaves the room
[15:52:57] Maas joins the room
[15:57:35] arjan leaves the room
[16:10:59] <Kaos> is there a sql sanitizer some where?
[16:11:36] <Kaos> for sanitizing a search string before sending it to the db..
[16:32:43] <Marc Worrell> yep - mod_search:to_tsquery will make it into a ts query for you.
[16:33:11] Maas leaves the room
[16:34:59] Kaos leaves the room
[16:35:28] Kaos joins the room
[16:54:56] Maas joins the room
[17:25:54] Maas leaves the room
[17:27:35] Maas joins the room
[17:36:45] Maas leaves the room
[18:24:17] <Kaos> so, passing text in parameters, using $N syntax is also safe?
[18:24:53] <Kaos> (as that's what to_tsquery is using on the input)
[20:25:54] arjan joins the room
[20:49:13] arjan leaves the room
[20:49:20] arjan joins the room
[20:51:05] arjan leaves the room
[20:51:41] arjan joins the room
[21:01:00] <z-bot> [blaaa] Hi around when can I expect a release package after 0.9.2? I might want to make a package for arch Linux, but I do not like packaging from git
[21:04:06] arjan leaves the room
[21:07:10] Maas joins the room
[21:08:18] Maas leaves the room
[21:10:19] <z-bot> [blaaa] An I have issues getting 0.9.2 to build with the recent Erlang. The got stuff from some time ago worked all right
[21:11:46] Maas joins the room
[22:01:39] <Kaos> Hi blaaa, I don't know how the release plans are. Try and get hold of Arjan, as he's usually the one making the releases, see if he has any plans for the near future..
[22:12:59] Maas leaves the room
[22:23:21] Maas joins the room
[22:33:30] cillian.deroiste joins the room
[23:32:38] cillian.deroiste leaves the room
[23:35:35] cillian.deroiste joins the room
[23:40:32] cillian.deroiste leaves the room
Powered by ejabberd Powered by Erlang Valid XHTML 1.0 Transitional Valid CSS!