Re: How to move on? #TYPO3

Posted: December 22nd, 2012 | Author: | 1 Comment »

As I wanted to answer Robert’s post but didn’t like the privacy of our Core-internal list, here’s some kind of response to it. Alongside I’ll try to explain the current situation and problems a bit***.

What happend?

In his post Robert kind of gave up his attempts to establish a TYPO3 product board [1]. The reasons for this are very wide-spread and mainly the various flame-wars in the last year and the very personal attacts brought him to the conclusion to:

* no longer invest my time into setting up or participating in an overall product team
* refrain from trying to establish leadership for the TYPO3 project
* concentrate on Flow and Neos and invite teams to participate in frequent meetings about it
* unsubscribe from the core internal mailing list

Even thought this shortens his entire mail a bit – that’s the bottom line it comes down to****.

Looking at the recording of the “Not-the-product-board” [7] meeting from last Tuesday [2] it seems that these steps aren’t necessary, as everybody seemed to be happy with the setup. So what’s the criticism actually about? Lucky enough I don’t have to describe it myself, but can cite an (again internal) response to an earlier mail from one of my fellow Core Team members:

Suddenly, when you (Robert) left the steering committee of the T3A because of the
new bodies and bylaws, you realized that there is no more power for you
to decide about things. That’s why you brought up the concept of a
product team. It’s an attempt to replace the former steering committee,
which has backed up the decisions of the core team in a nice but
undemocratic way.

In addition to these two standpoints we’ve a mixture of opinions and directions in our community and all of them cause quite some irritation [3,4,5]. With all these repeating fights and discussions Robert and (I guess) most of us ask ourselves how the community could avoid the fighting?

So what’s my opinion:

What we should avoid

  • Closed door meetings and private discussions – most of the “emotional outbursts” we saw where caused when the seemingly final results where presented to the community out of the blue. The fact that Core-internal is still a vivid place is a problem and we all should be abandon it.
  • Working without vision – the long holding support (since 2006) for Neos and the nice drive gridelements got from it’s fans [9] are two good examples that a vision can move mountains. But we should renew the vision statement and provide roadmaps on how we want to achieve it – mainly to motivate contributors and to enable collaboration between the teams.
  • Leaders or leader groups – other OpenSource products have their benevolent dictator – this wouldn’t work for our community. Their focus will never reach the entire community and in the end their decisions will always cause confusion. Imho Kasper made a great job to bring TYPO3 to life – but he/we saw that the “dictatorship” didn’t scale in the end [12] and imho Robert’s response and the controversy around the product board kind of shows that too.**
  • Personal attacks – nobody joined the TYPO3 community to fight, so there’s no reason to fight back. Within all controversy and disappointment everyone tries to improve TYPO3 in his way [10]

What we need

  • Diverse groups from all parts of the community – there’s no other way to capture all ideas and to gather various people from the entire community. As a nice side effect they make the “surface” of our community much larger and might help to involve new users. Btw. our community manager Ben van’t Ende will be happy to help kickstarting and coordinating groups.
  • Open meeting protocols and discussions – to make sure there’s a chance for everyone to catch-up later and to avoid bad surprises as we saw during the rebranding [3] or during the version schema change [6] In both cases small groups of people made major (not necessarily bad) decisions on their own behind closed doors – the following controversies showed that good intentions can turn out very negative.
  • Clear communication structures in the groups – to make sure things are “presented” appropriately. As see in [6] or in more recent situations [8] things would improve if groups had clear communication-channels. In [6] and [8]. seemingly official messages turned out to be personal attempts instead without actual “approval” / “consensus” from the related groups.
  • Rough product roadmaps – They should exist to make sure people share a common vision but they should not be too straight to make sure nobody feels too bound to it. They should also exist to enable some kind of measurement. We should be able to ask the Neos or CMS team whether they “live” along their vision or whether it would be better to adjust it.
  • Constructive honesty – people should discuss openly, in a respectful way [11]. Technical doubts and constructive criticism should be allowed, but it should be fair.

What does that mean for our products?

Honestly: I don’t know. Looking at the current situations with the lists from above: we have tons of groups working in many directions, we have a group that agreed on a common product-vision (two times) [13], but the group communicates behind closed doors. I’m not aware of any Neos or Flow roadmaps, but TYPO3 CMS has at least a (short) roadmap [14]. The Core team doesn’t have clear communication structures yet and as shown this raises ton’s of confusion. In addition our discussions tend to get very personal. It seems that sth. like a “product board” could help but…

Product board and leadership

On one hand I don’t like the way how the “product board” was positioned in the beginning – it should not lead anything or decide anything. On the other hand it’s great to have a group of people taking care to formulate a vision. The access to this group should be open to everyone, not just group leaders. Inner-circles, Top-10 groups, leader groups, Core-internal discussions should be avoided and open group-”setups” should be emphasized.

Hope this made sense?

Read on:*

[1] – Google Doc: Product Board (or “Product Team”)
[2] – TYPO3 Product Hangout (On Air)x
[3] – Die Marke TYPO3 erfindet sich neu
[4] – Rebranding: Get the green back
[5] – Wieviel Kommunikation und Roadmaps braucht ein Open Source Projekt?
[6] – TYPO3 6.0 at the corner? How is it possible?
[7] – @kdambekalns: Now a first #TYPO3 “product …” meeting, not official, no decisions, nothing.
[8] – @WrYBiT: @benvantende .. I expected, a mail by you and a news on T3O about it…
[9] – Startnext: Verbessertes TYPO3-Backend mit neuen Features
[10] – @tom_noice: . @thomas_hempel The discussions are there because so many people care.
[11] – Community Code of Conduct
[12] – King for a day, but not for a lifetime
[13] – The Phoenix team reports on the Developer Days 2012
[14] – Proposal for the upcoming Roadmap and LTS

* take your time to read the endless lines of comments.
** Imho Oliver Hader choose a good attempt to (not)”lead” the Core Team – more in the sense of “managing” and “enabling” without ruling
*** Some tweets and messages from the Core members might have been quite confusing without the context
**** I’d prefer not to play TYPO3-leaks here, so it’s up to Robert and the others to publish their mails by themselfs

TYPO3 login state and Varnish cache

Posted: December 10th, 2012 | Author: | Tags: , , | Comments Off

Caching is hard in complex page setups with user specific content, especially when public pages change their content once a user is logged in. TYPO3 is smart enough to deal with the login state properly and cache appropriately. Once Varnish is involved, it’s quite tricky to cache as much as possible without loosing the dynamic content. But it’s not impossible and here’s my summary how we resolved it for typo3.org.

Setup

The basic Varnish setup is more or less always the same and best described by Farbrizio Branca. On top of that we need some TypoScript parameter tweaking to get the cache-control-headers in TYPO3 straight – Daniel Pötzinger’s article covers them best. Another very handy thing which can be found in Fabrizio’s blog is the simplified flow chart for the various Varnish subroutines.Based on that all pages should be cached properly and your site should run smoothly. But in case you have a page with personalized content, you’ll have to reconsider some parts.

Problem

The event submission page on typo3.org is a good example. In case the user is not logged in (a.k.a public page), we just want to show a message which guides him to the login. If there’s a login active (a.k.a user page), we’ll show the submission form instead. In both cases we could cache the content nicely, but how would we ensure that Varnish delivers the correct content?

Solution

Ajax could be a solution, but for large sites it’s usually smarter to avoid as much JavaScript as possible. EdgeSideIncludes (ESI) are another option, but I agree with Daniel, they’re not really useful in this case and I’d rather go with Ajax than with ESI.

What we want in this scenario, is to cache the public page in Varnish and pass to the user page generated by TYPO3 if we find that the user is logged in. But this should of course only happen on pages where this is really necessary – normal pages should just ignore the login state of the user. Therefore we need sth. to distinguish normal from login specific pages in Varnish. Lucky enough TYPO3 already provides a field in the pages properties which allows this distinction. Using the Login Behaviour (pages.fe_login_mode) field, you can enable and disable the user-login for specific branches and pages*. As we want to whitelist login specific pages, our root page should have the default setting “Disable Login” – this will be inherited to all sub-pages. All the login specific pages should have the setting “Re-Enable login”.

Once this is done, we need a way to carry that out to Varnish. We improved EXT:cacheinfo for that purpose, with that it now carries a “loginAllowedInBranch” or “noLoginAllowedInBranch” value in the “X-T3CacheInfo” header. Using all that, the Varnish VCL can be extended to make use of it like this:

sub vcl_hit {
  if (obj.http.X-T3CacheInfo ~ "loginAllowedInBranch") {
    set obj.http.Cache-Control = "private";
    if (req.http.Cookie ~ "(e_typo_user|PHPSESSID|_pk_.*)") {
      # Do not cache requests which come from a logged in user
      return (pass);
    }
  }
}

This is straight forward. For every page which allows logins, we make sure that the client does not keep them in his cache. In case we’re actually on such a page and find the related login-cookies, we pass the request along to TYPO3, otherwise we deliver the public page right away from the cache. The fact that we pass the request along to TYPO3 in some cases doesn’t mean that we’ll deliver the user page, it just indicates that we’ve to rely on TYPO3 to make the right choice based on the actual login state.

Conclusion

For me the beauty here lies in the simplicity. Once you managed to wrap your head around the flow chart and once you managed to deliver appropriate meta-data to Varnish, many more complex scenarios can be resolved equally.

As most of the typo3.org stuff, this solution came from a great team. In this case Michael Stucki and Daniel Pötzinger helped to craft the final solution – thanks guys :)

* the naming of the field’s labels is really irritating – especially “0 – Enable login” should be “Inherit setting” as it really does not force any setting.

Finding TypoScript errors.

Posted: February 16th, 2012 | Author: | Tags: , | 10 Comments »

When you work on TypoScript templates in TYPO3, errors might show up in the TypoScript Object Browser. Within the error messages you’ll see a more or less detailed error description with the related line number. Within most setups these line numbers won’t relate to any of your sys_template records or TypoScript files directly. But they still provide value if you know how they help to find the right spot. As it’s not too obvious how to find the right spot I’ve created a little screenshot series to guide you to the broken spots in your templates.

So that’s what you might see in your TypoScript Object Browser:

The error message from your TypoScript Object Browser might look like this.

 Switching from there to the Template Analyzer:

Switching to the TypoScript analyzer

At the bottom of the Template Analyzer, you’ll find a “Complete TS” section and a link ~which looks like normal text.

And you'll find a link to the fully concatenated TypoScript of your current page.

Clicking on that link will give you the entire concatenated TypoScript and here you’ll also find that the line numbers finally match to the error message.

And you'll find that the line numbers are now what you saw in the error message before (might need some scrolling).

A well hidden gem which works most likely in all TYPO3 4.x versions ;)

Edit: In the meantime, Ingo’s patch made it through the review process. So users of TYPO3 4.7 and above will find a nice and handy “Show details” link next to the error message. Makes it much much faster to find the broken spot. Thanks Ingo :) / (@irnnr)

Visualizing TYPO3 Core activity

Posted: January 20th, 2012 | Author: | Tags: , | 3 Comments »

DISCLAIMER: Don’t take the following too serious – when reading this post please keep in mind that the TYPO3 community itself consists of much more than just the Core team activity – many things take place outside of code repositories and can’t be measured anyhow. Every contribution is important and every single action has value.

Using Gerrit sometimes feels quite lonesome – you don’t really see who’s active and you don’t really get a feeling on how much is done in the TYPO3 Core at a certain point. To visualize on how active the contributors are and to show the most active members I applied a little scoring and summed up the results for the time since 2006.

Snippet of the impact chart.

Early 2006 impact chart with few early contributers - click the image to see the full chart

The scoring is quite easy – every author of a patch gets 10 points, testers get 3 and reviewers 1 point*. Looking at the stats it seems that even the statistics pulled from the old Subversion days seem to meet up with today’s numbers – of course we’ve to keep in mind that everything which was pulled from Subversion doesn’t really point to the author but to the actual committer (except if there was a “Thanks to XXXXX” reference in the commit).

To visualize the numbers I choose a Github like impact chart**. Each contributor has it’s own color and line in it and whenever he got active the width of the line scales up. To maintain the overview the scale of the width isn’t linear and every contributor who’s not within the “Top 20″ had to be scaled down to “1″. The line stops if the contributor didn’t get active anymore. The scores are grouped and compared by month.

Snapshot of the "Top 20" stats taken Jan. 14th 2012

Taken the last 3months - see the entire stats on the referenced page.

In addition to the chart I also created a table based overview for the “Top 20″ contributors with their score***.

Few things I got from the numbers. There have been 290 contributors already – simply amazing. Comparing the (huge) scores of the release managers with their fellow contributors shows once more that they can be really proud about the job they did or do. It’s also quite cool to see that many people keep sticking around. And finally it was quite surprising to see that our community manager Ben van’t Ende can also be found in the stats. Besides that it’s up to everyone else to find their conclusions from these numbers. I hope it’s motivating everyone to see that the community is active as always.

The script I created to generate the charts isn’t too nice at the moment, but I promise to publish it once I cleaned it up a bit. If you can’t wait to get your hands on it, feel free to send me a mail or tweet. – but I published it anyways: github.com/tolleiv/Repo-Activity-Monitor

Links in short: Impact chart, Monthly “Top 20″ tables.

* The scoring includes the commits to the Core master and all related submodule commits. Unfortunately the submodule commits don’t hold the reviewer and tester information. Also some of the latest changes made in the submodules may not show up at the moment because the submodule pointers haven’t been updated yet.
** Inspired by the incredible RaphaëlJS vector graphics library which is distributed with an MIT license and can be found on raphaeljs.com
*** Again – don’t take it too serious – the scoring doesn’t take into account that some patches can be written in 15 minutes where others take days. It also only uses the final testers and reviewers mentioned in the commit message, other contributions are not counted atm

TemplaVoila 1.6.x

Posted: January 16th, 2012 | Author: | Tags: , | 3 Comments »

Quite some things changed in the past months and I never found the time to clear up my mind and write up a summary.

First of all TemplaVoila 1.6 was released parallel to TYPO3 4.6. It shipped with 22 bug and compatibility fixes. In general the 1.6.x branch is supposed to be compatible with TYPO3 4.4+ which also made it possible to clean up the extension quite a bit. In addition to that TemplaVoila 1.6.1 will show up in the TER in the next few hours. It fixes 10 additional issues.

Besides that, the main repository for TemplaVoila was moved to git.typo3.org and the old Subversion repository and my Github repository have been removed. The new repository location also enables and enforces a new way to contribute code changes for the project. Comparable to the TYPO3v4 Core every change request can be sent to Gerrit, where I can review the changes before they get merged into the repository. That workflow turns out to be very efficient for me. A detailed summary on how to contribute code to any repository hosted on git.typo3.org can be found on wiki.typo3.org. To sum up some of the steps – here’s how you submit a patch*:

git clone git://git.typo3.org/TYPO3v4/Extensions/templavoila.git
cd templavoila
scp -p -P 29418 <username>@review.typo3.org:hooks/commit-msg .git/hooks/
git checkout -b workingBranch
# ... work on the files ...
git add <changedFile>
git commit
git push origin HEAD:refs/for/master/<topic>

A further general change to the team happend more or less silently. During 2011 nobody from the old team or any new developer showed interest in the project and only few contributors helped with bugfixes or reviews. Due to that I also changed my attitude regarding future releases.
I’m currently planning to release one version parallel with every main TYPO3 4.x release, to make sure that TemplaVoila works in new versions and to make sure people can enjoy TYPO3 with TemplaVoila in the future as well. I’ll also try to keep it compatible with all “stable” releases and aim to keep the issue count within the bugtracker as low as possible.
But I’m not planning to integrate new major features such as the field content sliding (use EXT:kb_tv_cont_slide please) or a context sensitive content wizard (a.k.a “content firewall”). Also major refactorings won’t happen because they’d consume far too much of my time – therefore also the desperately needed update of the mapping module or a new new page module will not be implemented in the near future.

Nevertheless I’m open for any type of contribution and I’d be happy to review and test any patch showing up in Mantis or Gerrit – I’m just not able to spent a major part of my freetime for it.

* You have to sign the Contributor License Agreement to be able to push any change – it can be found on typo3.org

« Older Entries