Tag Archives: Web Development

How to Use New Relic Custom Dashboards & Why You’d Want To

Today we’re going to look at New Relic custom dashboards. Specifically, I will show you three ways that I tend to use custom dashboards:

  • creating an overview dashboard from existing charts
  • creating your own charts from existing data captured by New Relic
  • creating dashboards for your custom metrics
Sponsored ContentThis content was commissioned by New Relic and was written and/or edited by the Tuts+ team. Our aim with sponsored content is to publish relevant and objective tutorials, case studies, and inspirational interviews that offer genuine educational value to our readers and enable us to fund the creation of more useful content.

But, before we get into any of that, we must first ask ourselves a question, why should we use custom dashboards at all? New Relic does a pretty good job of presenting your application performance data. Just using the regular UI, you can glean way more information about the performance of your app than you ever could have before you started using New Relic. So, before I show you how to use custom dashboards, I’ll explain why I think anyone using New Relic should look at them sooner rather than later.

Why Use Custom Dashboards at All?

It’s true, most people happily use New Relic without ever looking at the custom dashboard functionality that it provides. It’s not until you’re a fairly advanced user that you may decide to look at custom dashboards and see what they have to offer you. I think this is a shame, playing around with custom dashboards can not only allow you to slice and dice your app performance data in different ways, but can also:

  • teach you quite a bit about how New Relic captures metrics
  • allow you to learn exactly what kind of data gets stored for the metrics that get captured
  • teach you about the limitations of New Relic charts

You can learn all those things by reading the documentation, but playing around with custom dashboards, allows us to begin to understand some of these things by example (on a more intuitive level), rather than just knowing it as a bunch of dot points. As is often the case with technology, tinkering with an unrelated area of a tool, will sometimes give you more insight and understanding into how the more commonly used areas of the same tool work. Using custom dashboards will make you a more savvy New Relic user, and if you’ve read the other New Relic posts that I’ve written, you’ll know how I feel about knowing your tools.

Creating an Overview Dashboard from Existing Charts

The one custom dashboard I always like to build is what I call ’24 hours at a glance’. I take a bunch of existing charts that I consider important for a single application, lock the time period to the last 24 hours and put them all together on one dashboard. Whenever I am looking at a specific application in New Relic, this will be the first screen I look at to see if there is anything particularly bad that jumps out at me from the last 24 hours. Let’s see how we can build this kind of dashboard.

Firstly, we need to create a new custom dashboard. In the New Relic UI click onDashboards->Create custom dashboard. On the next screen, we give our dashboard a name (you could use “24 hours at a glance”, or your own name) and pick the grid layout. Essentially, grid layout is a collection of charts all the same size and overview layout is one big chart surrounded by a bunch of smaller charts (we will use overview layout in the next section).

01_create_dashboard

Now we need to select the app for which we want to create our dashboard and then find some relevant charts to add. I like to add the following:

  • server average response time
  • historical server average response time
  • browser average response time
  • error rate
  • throughput
  • historical throughput
  • application CPU usage by host
  • application memory usage by host
  • top five web transactions by wall clock time
  • downtime
  • top countries by throughput
  • top five database operations by wall clock time

Depending on your application you may want to add some others, but I find this gives me a fairly good snapshot of what’s happening with the application and what to investigate if anything is wrong. Let’s find one of these charts and add them to our new dashboard. The server average response time is an easy one, as it’s the big chart in the Overview section of the Monitoring tab for an application. Each chart in the New Relic UI has a little New Relic logo in the bottom right corner, when you hover your mouse over this logo it turns into a plus sign, clicking on the plus will allow you to add this chart to a dashboard:

02_add_chart_to_dashboard

But before we add our chart, we need to change the time window for the New Relic UI to be 24 hours. We need to do this in order to give ourselves the ability to ‘lock’ the chart to the last 24 hours when we actually add it to the dashboard (this is awkward UXin my opinion, but at least we have a way to do what we need):

03_change_time_window

We can now go ahead and add the chart:

04_add_chart_to_dashboard

Don’t forget to tick the Lock to span checkbox. When we now visit our new dashboard, the chart we’ve just added should be there:

05_dashboard_with_one_chart

We can rinse and repeat the above process until we’ve added all the charts that we want. In the end, it should look something like this:

06_full_dashboard

You can click the Edit dashboard button in the top right corner which will let you drag the charts around and arrange them in the order you want. The only thing to note is that you can’t modify the individual charts in any way (for example, you may have wanted to have a more descriptive chart title, but you can’t change it) since they are standard New Relic charts.

The other custom dashboard I always like to build from existing charts is the ‘All applications at a glance’. This is only applicable if you have multiple applications you’re looking after. Here we pick one or two of the most important charts for every relevant application and put them together. It’s usually a safe bet to use the ‘Response Time’ chart from each application. The actual process of putting the dashboards together is the same as described above, you’ll just need to switch applications to get the relevant charts from each. In the end, you should end up with something like this:

07_all_applications

This is the screen I’ll tend to look at first, when I log in to New Relic. It may be useful to lock the time of each chart to 24 hours just like we did for our ’24 hours at a glance’ dashboard, but that’s up to you. Of course, this is only relevant if you’re supporting multiple applications. Having said that, if you have multiple pre-production environments for your application (for staging or load test), you may want to put them all together into a dashboard similar to this one, it may help you catch changes that degrade performance before the code ends up in production.

Creating Your Own Charts from Existing Data

The New Relic UI suffers from some necessary limitations. It needs to be all things to all people, so they can only group together charts and tables that would make sense for all web applications. Most of the time the UI screens will limit you to looking at one transaction at a time and one or two sets of metrics at a time, if you need to access others you have to click around. The good news is, with custom dashboards, this limitation no longer applies. We know which transactions are related within our domain, we also know which metrics are important to us on a per transaction basis. We can build a dashboard that groups together several related transactions with all the important metrics for each and look at it on the one screen.

Let’s say we have a particularly important transaction in our application, it might make sense to have a dashboard where we can see most of the important information about this transaction at a glance. Here on Tuts+ we have a concept of articles (obviously) and articles are pretty important to us, let’s build a dashboard to keep an eye on them.

Once again we need to create a new dashboard just like before, we’ll call it ‘Article overview’, but this time we’ll use an overview layout. We don’t have to go hunting for charts as we’ll be creating our own custom charts, so click the big button to create the main chart for our dashboard:

08_add_custom_chart

It will ask you if you want to add a chart or a table, we will add a table later, for now choose chart. You will see a screen that looks like this:

09_new_chart

The main thing to look at here is the metric that you want to display. When you click inside the metrics text-box it will drop down a list of top level metrics that you can select. The metrics in New Relic are named as prefix/category/label. In the case of a Railsapp, the prefix might be Controller or ActiveRecord (if you’re not using Rails, the prefix for transactions will be WebTransactions). For Controller, the category would be the name of the controller and the label will be the action name. If you explore some of the metrics while you’re playing around with your first chart, you will start to have a feel for the kind of metrics you have access to and where to find them. If you don’t see the metrics you expect, make sure that you have the right application selected within New Relic, this always trips me up.

Back to our main chart. The metric we’re after will be tied to our ArticlesController, so its name is Controller/articles/show. Once we’ve picked the metric, the contents of the Value drop-down will change to contain all the values that make sense for this metric. It’s worth once again exploring all the various values and seeing what the resulting chart actually contains. In our case, “Average response time” seems like a good thing to have as our main chart.

At this point, if we give our chart a title and click the preview button we can see what it looks like:

10_articles_preview

This looks OK, but I want the Y axis to be in milliseconds and I want the units on the axis as well. So, let’s drop down the advanced options for the chart and change the number format to be ‘To Milliseconds‘, we will also set the Y-axis unit label as ‘ms‘:

11_filled_out_main_chart_form

Our chart now looks good on the preview. The only thing of note that we haven’t talked about is the Chart click-through drop-down. This essentially allows your chart to be a link to another custom dashboard, when you click on the chart, that dashboard will be displayed. We don’t need this functionality, so we leave the drop-down alone. We’ll now go ahead and save our chart.

12_completed_chart_preview

We can now add the various sub-charts. In our case, I happen to know that Tuts+ has a concept of tutorials (also, obviously) which is closely related to articles, so if I am going to keep an eye on article average response time, it’s probably a good idea to have a tutorial average response time close by as a comparison, so I will create a chart for that. We’ll follow the same steps as above, in the end our dashboard will look like this:

13_article_and_tutorial_dashboard

Hmm, it looks like the average response time for articles is much higher than tutorials, but I happen to know that both of them share a significant amount of code, strange. But it’s also an indication that our custom dashboard is already paying dividends and we haven’t even finished constructing it yet. I could have found this information by looking it up in the regular New Relic UI, but having the charts side by side like this helps to really bring home the fact that there might be an issue.

It might also be good to see where our ArticlesController sits compared to other controllers, as far as their maximum response time goes, this is a job for a table. We add another chart just like before, but this time pick table instead of chart. To create tables with multiple rows, we need to use wildcards in our metric name. In our case, I will set the metric to be Controller/, this will select all the metrics under the*Controller prefix, I will now set the limit text-box to be 10 which will do exactly as you expect and set the number of rows in our table to ten. In the end, we should have something that looks like this, right before we save:

14_table

Our custom dashboard will now be:

15_custom_dashboard_with_table

It seems that ArticlesController#show has the longest maximum response time out of all the controller actions, including TutorialsController#show, this is very curious and I should probably make a note to look into this.

We can continue adding a bunch of other charts such as end user average response time or calls per minute. However, some things you just can’t construct using a custom chart, such as historical throughput or response time. Luckily we can always fall back on finding the charts that we want somewhere else in New Relic and simply adding those ones to our custom dashboard.

The only limitation with using a dashboard which has custom charts is that you have to have the correct app selected within New Relic, otherwise all the custom charts on your dashboard will be blank.

Creating Dashboards for Your Custom Metrics

If you read my last article on custom metrics, you might remember me mentioning that the only way for you to view the custom metric data that you’ve collected is to create a custom dashboard in New Relic, this is the third reason to use custom dashboards. If you collect a lot of custom metrics, this might be the best reason of all.

On Tuts+ we have a concept of categories (yet again, obviously), I just happen to know that we have a few custom metrics floating around for categories. Let’s see if we can put these on a dashboard and actually get an idea of what’s happening. We’ll create another dashboard and call it ‘Categories Custom Metrics’. All custom metrics in New Relic should live under the Custom prefix and this is where we find the metrics we’re looking for:

16_custom_metrics

We’ll create a couple of charts, one to see how long building presenters inCategoriesController takes and the other to see how long it takes to get a link hash from the presenters. The key thing to know with custom metrics, is what kind of data you’re actually sending to New Relic. In this case I happen to know that we’re measuring time, so I can pick ‘Average value‘ as my metric value and set the Number format tomilliseconds to get a reasonable looking chart. After creating both charts, our custom dashboard looks like this:

17_dashboard_for_custom_metrics

It looks like getting the link hash from the presenters is very quick and doesn’t fluctuate too much, there is no need to optimize anything here and I can probably stop collecting this metric completely (no need to capture unnecessary data). However, building the presenters takes significantly more time, we can look into this further to see if it can be optimized. We can also keep an eye on the chart (by looking at the custom dashboard occasionally) to make sure the performance doesn’t degrade as we continue to work on the application.

Conclusion

Custom dashboards are not a panacea. Aside from creating charts for custom metrics, you can do everything that custom dashboards can do with the regular New Relic UI. However playing with custom dashboards will definitely help you become more of a power user of New Relic, with a deeper understanding of how it works under the hood. In addition, the ability to look at your performance metrics in different ways can be an invaluable tool to help you surface potential performance problems before they have a significant impact on your application.

If you have any questions about New Relic custom dashboards, don’t be afraid to leave a comment and I’ll do my best to answer. Also, if you’ve used custom dashboards to good effect in the past, please share any tips you might have, it’s always interesting to see how other people use their tools.

Setting Up a Local Mirror for Composer Packages With Satis

Installing all your PHP libraries with Composer is a great way to save time. But larger projects automatically tested and run at each commit to your software version control (SVC) system will take a long time to install all the required packages from the Internet. You want to run your tests as soon as possible through your continuous integration (CI) system so that you have fast feedback and quick reactions on failure. In this tutorial we will set up a local mirror to proxy all your packages required in your project’scomposer.json file. This will make our CI work much faster, install the packages over the local network or even hosted on the same machine, and make sure we have the specific versions of the packages always available.

 

What Is Satis?

Satis is the name of the application we will use to mirror the various repositories for our project. It sits as a proxy between the Internet and your composer. Our solution will create a local mirror of a few packages and instruct our composer to use it instead of the sources found on the Internet.

Here is an image that says more than a thousand words.

Architecture

Our project will use composer as usual. It will be configured to use the local Satis server as the primary source. If a package is found there, it will be installed from there. If not, we will let composer use the default packagist.org to retrieve the package.

Getting Satis

Satis is available through composer, so installing it is very simple. In the attached source code archive, you will find Satis installed in the Sources/Satis folder. First we will install composer itself.

1
2
3
$ curl -sS https://getcomposer.org/installer | php
#!/usr/bin/env php All settings correct for using Composer Downloading...
 Composer successfully installed to: /home/csaba/Personal/Programming/NetTuts/Setting up a local mirror for Composer packages with Satis/Sources/Satis/composer.phar Use it: php composer.phar

Then we will install Satis.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ php composer.phar create-project composer/satis --stability=dev --keep-vcs Installing composer/satis (dev-master eddb78d52e8f7ea772436f2320d6625e18d5daf5)
  - Installing composer/satis (dev-master master)
    Cloning master
 Created project in /home/csaba/Personal/Programming/NetTuts/Setting up a local mirror for Composer packages with Satis/Sources/Satis/satis Loading composer repositories with package information Installing dependencies (including require-dev) from lock file
  - Installing symfony/process (dev-master 27b0fc6)
    Cloning 27b0fc645a557b2fc7bc7735cfb05505de9351be
  - Installing symfony/finder (v2.4.0-BETA1)
    Downloading: 100%
  - Installing symfony/console (dev-master f44cc6f)
    Cloning f44cc6fabdaa853335d7f54f1b86c99622db518a
  - Installing seld/jsonlint (1.1.1)
    Downloading: 100%
  - Installing justinrainbow/json-schema (1.1.0)
    Downloading: 100%
  - Installing composer/composer (dev-master f8be812)
    Cloning f8be812a496886c84918d6dd1b50db5c16da3cc3
  - Installing twig/twig (v1.14.1)
    Downloading: 100%
 symfony/console suggests installing symfony/event-dispatcher () Generating autoload files

Configuring Satis

Satis is configured by a JSON file very similar to composer. You can use whatever name you want for your file and specify it for usage later. We will use “mirrored-packages.conf“.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
{
    "name": "NetTuts Composer Mirror",
    "homepage": "http://localhost:4680",
    "repositories": [
        { "type": "vcs", "url": "https://github.com/SynetoNet/monolog" },
        { "type": "composer", "url": "https://packagist.org" }
    ],
    "require": {
        "monolog/monolog": "syneto-dev",
        "mockery/mockery": "*",
        "phpunit/phpunit": "*"
    },
    "require-dependencies": true
}

Let’s analyze this configuration file.

  • name – represents a string that will be shown on the web interface of our mirror.
  • homepage – is the web address where our packages will be kept. This does not tell our web server to use that address and port, it is rather just information of a working configuration. We will set up the access to it on that addres and port later.
  • repositories – a list of repositories ordered by preference. In our example, the first repository is a Github fork of the monolog logging libraries. It has some modifications and we want to use that specific fork when installing monolog. The type of this repository is “vcs“. The second repository is of type “composer“. Its URL is the default packagist.org site.
  • require – lists the packages we want to mirror. It can represent a specific package with a specific version or branch, or any version for that matter. It uses the same syntax as your “require” or “require-dev” in your composer.json.
  • require-dependencies – is the final option in our example. It will tell Satis to mirror not only the packages we specified in the “require” section but also all their dependencies.

To quickly try out our settings we first need to tell Satis to create the mirrors. Run this command in the folder where you installed Satis.

1
$ php ./satis/bin/satis build ./mirrored-packages.conf ./packages-mirror Scanning packages Writing packages.json Writing web view

While the process is taking place, you will see how Satis mirrors each found version of the required packages. Be patient it may take a while to build all those packages.

Satis requires that date.timezone to be specified in the php.ini file, so make sure it is and set to your local timezone. Otherwise an error will appear.

1
2
3
[Twig_Error_Runtime]
  An exception has been thrown during the rendering of a template
("date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set) function.

Then we can run a PHP server instance in our console pointing to the recently created repository. PHP 5.4 or newer is required.

1
2
3
$ php -S localhost:4680 -t ./packages-mirror/ PHP 5.4.22-pl0-gentoo Development Server started at Sun Dec  8 14:47:48 2013 Listening on http://localhost:4680 Document root is /home/csaba/Personal/Programming/NetTuts/Setting up a local mirror for Composer packages with Satis/Sources/Satis/packages-mirror Press Ctrl-C to quit.
[Sun Dec  8 14:48:09 2013] 127.0.0.1:56999 [200]: /
[Sun Dec  8 14:48:09 2013] 127.0.0.1:57000 [404]: /favicon.ico - No such file or directory

And we can now browse our mirrored packages and even search for specific ones by pointing our web browser to http://localhost:4680:

MirrorWebpage

Let’s Host It on Apache

If you have a running Apache at hand, creating a virtual host for Satis will be quite simple.

1
2
3
4
5
6
7
8
9
Listen 4680
<Directory "/path/to/your/packages-mirror"> Options -Indexes FollowSymLinks AllowOverride all Order allow,deny Allow from all
</Directory>
<VirtualHost *:4680> DocumentRoot "/path/to/your/packages-mirror" ServerName 127.0.0.1:4680 ServerAdmin admin@example.com
 ErrorLog syslog:user
</VirtualHost>

We just use a .conf file like this, put in Apache’s conf.d folder, usually/etc/apache2/conf.d. It creates a virtual host on the 4680 port and points it to our folder. Of course you can use whatever port you want.

Updating Our Mirrors

Satis can not automatically update the mirrors unless we tell it. So the easiest way, on any UNIX like system, is to just add a cron job to your system. That would be very easy, and just a simple script to execute our update command.

1
2
3
#!/bin/bash php /full/path/to/satis/bin/satis build \
/full/path/to/mirrored-packages.conf \
/full/path/to/packages-mirror

The drawback of this solution is that it is static. We have to manually update themirrored-packages.conf every time we add another package to our project’scomposer.json. If you are part of a team in a company with a big project and a continuous integration server, you can’t rely on people remembering to add the packages on the server. They may not even have permissions to access the CI infrastructure.

Dynamically Updating Satis Configuration

It’s time for a PHP TDD exercise. If you just want your code ready and running, check out the source code attached to this tutorial.

1
2
3
4
5
6
require_once __DIR__ . '/../../../../vendor/autoload.php';
 class SatisUpdaterTest extends PHPUnit_Framework_TestCase {
    function testBehavior() {
        $this->assertTrue(true);
    }
}

As usual we start with a degenerative test, just enough to make sure we have a working testing framework. You may notice that I have quite a strange looking require_once line, this is because I want to avoid having to reinstall PHPUnit and Mockery for each small project. So I have them in a vendor folder in my NetTuts‘ root. You should just install them with composer and drop the require_once line altogether.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
class SatisUpdaterTest extends PHPUnit_Framework_TestCase {
    function testDefaultConfigFile() {
        $expected = '{
    "name": "NetTuts Composer Mirror",
    "homepage": "http://localhost:4680",
    "repositories": [
        { "type": "vcs", "url": "https://github.com/SynetoNet/monolog" },
        { "type": "composer", "url": "https://packagist.org" }
    ],
    "require": {
    },
    "require-dependencies": true
}';
        $actual = $this->parseComposerConf('');
        $this->assertEquals($expected, $actual);
    }
}

That looks about right. All the fields except “require” are static. We need to generate only the packages. The repositories are pointing to our private git clones and to packagist as needed. Managing those is more of a sysadmin job than a software developer’s.

Of course this fails with:

1
PHP Fatal error:  Call to undefined method SatisUpdaterTest::parseComposerConf()

Fixing that is easy.

1
2
private function parseComposerConf($string) {
}

I just added an empty method with the required name, as private, to our test class. Cool, but now we have another error.

1
PHPUnit_Framework_ExpectationFailedException : Failed asserting that null matches expected '{ ... }'

So, null does not match our string containing all that default configuration.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
private function parseComposerConf($string) {
    return '{
"name": "NetTuts Composer Mirror",
"homepage": "http://localhost:4680",
"repositories": [
    { "type": "vcs", "url": "https://github.com/SynetoNet/monolog" },
    { "type": "composer", "url": "https://packagist.org" }
],
"require": {
},
"require-dependencies": true
}';
}

OK, that works. All tests are passing.

1
PHPUnit 3.7.28 by Sebastian Bergmann. Time: 15 ms, Memory: 2.50Mb OK (1 test, 1 assertion)

But we introduced a horrible duplication. All that static text in two places, written character by character in two different places. Let’s fix it:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class SatisUpdaterTest extends PHPUnit_Framework_TestCase {
    static $DEFAULT_CONFIG = '{
    "name": "NetTuts Composer Mirror",
    "homepage": "http://localhost:4680",
    "repositories": [
        { "type": "vcs", "url": "https://github.com/SynetoNet/monolog" },
        { "type": "composer", "url": "https://packagist.org" }
    ],
    "require": {
    },
    "require-dependencies": true
}';
    function testDefaultConfigFile() {
        $expected =  self::$DEFAULT_CONFIG;
        $actual = $this->parseComposerConf('');
        $this->assertEquals($expected, $actual);
    }
    private function parseComposerConf($string) {
        return self::$DEFAULT_CONFIG;
    }
}

Ahhh! That’s better.

1
2
3
4
5
6
    function testEmptyRequiredPackagesInComposerJsonWillProduceDefaultConfiguration() {
    $expected = self::$DEFAULT_CONFIG;
    $actual = $this->parseComposerConf('{"require": {}}');
    $this->assertEquals($expected, $actual);
}

Well. That also passes. But it also highlights some duplication and useless assignment.

1
2
3
4
5
6
7
8
    function testDefaultConfigFile() {
    $actual = $this->parseComposerConf('');
    $this->assertEquals(self::$DEFAULT_CONFIG, $actual);
}
 function testEmptyRequiredPackagesInComposerJsonWillProduceDefaultConfiguration() {
    $actual = $this->parseComposerConf('{"require": {}}');
    $this->assertEquals(self::$DEFAULT_CONFIG, $actual);
}

We inlined the $expected variable. $actual could also be inlined, but I like it better this way. It keeps the focus on what is tested.

Now we have another problem. The next test I want to write would look like this:

1
2
3
4
5
6
7
function testARequiredPackageInComposerWillBeInSatisAlso() {
    $actual = $this->parseComposerConf(
    '{"require": {
        "Mockery/Mockery": ">=0.7.2"
    }}');
    $this->assertContains('"Mockery/Mockery": ">=0.7.2"', $actual);
}

But after writing the simple implementation, we will notice it requires json_decode()and json_encode(). And of course these functions reformat our string and matching strings will be difficult at best. We have to take a step back.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
function testDefaultConfigFile() {
    $actual = $this->parseComposerConf('');
    $this->assertJsonStringEqualsJsonString($this->jsonRecode(self::$DEFAULT_CONFIG), $actual);
}
 function testEmptyRequiredPackagesInComposerJsonWillProduceDefaultConfiguration() {
    $actual = $this->parseComposerConf('{"require": {}}');
    $this->assertJsonStringEqualsJsonString($this->jsonRecode(self::$DEFAULT_CONFIG), $actual);
}
 private function parseComposerConf($jsonConfig) {
    return $this->jsonRecode(self::$DEFAULT_CONFIG);
}
 private function jsonRecode($json) {
    return json_encode(json_decode($json, true));
}

We changed our assertion method to compare JSON strings and we also recoded our$actual variable. ParseComposerConf() was also modified to use this method. You will see in a moment how it helps us. Our next test becomes more JSON specific.

1
2
3
4
5
6
7
function testARequiredPackageInComposerWillBeInSatisAlso() {
    $actual = $this->parseComposerConf(
        '{"require": {
            "Mockery/Mockery": ">=0.7.2"
        }}');
    $this->assertEquals('>=0.7.2', json_decode($actual, true)['require']['Mockery/Mockery']);
}

And making this test pass, along with the rest of the tests, is quite easy, again.

1
2
3
4
5
6
7
8
private function parseComposerConf($jsonConfig) {
    $addedConfig = json_decode($jsonConfig, true);
    $config = json_decode(self::$DEFAULT_CONFIG, true);
    if (isset($addedConfig['require'])) {
        $config['require'] = $addedConfig['require'];
    }
    return json_encode($config);
}

We take the input JSON string, decode it, and if it contains a “require” field we use that in our Satis configuration file instead. But we may want to mirror all versions of a package, not just the last one. So maybe we want to modify our test to check that the version is “*” in Satis, regardless of what exact version is in composer.json.

1
2
3
4
5
6
7
function testARequiredPackageInComposerWillBeInSatisAlso() {
    $actual = $this->parseComposerConf(
        '{"require": {
            "Mockery/Mockery": ">=0.7.2"
        }}');
    $this->assertEquals('*', json_decode($actual, true)['require']['Mockery/Mockery']);
}

That obviously fails with a cool message:

1
PHPUnit_Framework_ExpectationFailedException : Failed asserting that two strings are equal. Expected :* Actual   :>=0.7.2

Now, we need to actually edit our JSON before re-encoding it.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
private function parseComposerConf($jsonConfig) {
    $addedConfig = json_decode($jsonConfig, true);
    $config = json_decode(self::$DEFAULT_CONFIG, true);
    $config = $this->addNewRequires($addedConfig, $config);
    return json_encode($config);
}
 private function toAllVersions($config) {
    foreach ($config['require'] as $package => $version) {
        $config['require'][$package] = '*';
    }
    return $config;
}
 private function addNewRequires($addedConfig, $config) {
    if (isset($addedConfig['require'])) {
        $config['require'] = $addedConfig['require'];
        $config = $this->toAllVersions($config);
    }
    return $config;
}

To make the test pass we have to iterate over each element of the required packages array and set their version to ‘*’. See method toAllVersion() for more details. And to speed up this tutorial a little bit, we also extracted some private methods in the same step. This way, parseComoserConf() becomes very descriptive and easy to understand. We could also inline $config into the arguments of addNewRequires(), but for aesthetic reasons I left it on two lines.

But what about “require-dev” in composer.json?

1
2
3
4
5
6
7
8
9
function testARquiredDevPackageInComposerWillBeInSatisAlso() {
    $actual = $this->parseComposerConf(
        '{"require-dev": {
            "Mockery/Mockery": ">=0.7.2",
            "phpunit/phpunit": "3.7.28"
        }}');
    $this->assertEquals('*', json_decode($actual, true)['require']['Mockery/Mockery']);
    $this->assertEquals('*', json_decode($actual, true)['require']['phpunit/phpunit']);
}

That obviously fails. We can make it pass with just copy/pasting our if condition inaddNewRequires():

01
02
03
04
05
06
07
08
09
10
11
private function addNewRequires($addedConfig, $config) {
    if (isset($addedConfig['require'])) {
        $config['require'] = $addedConfig['require'];
        $config = $this->toAllVersions($config);
    }
    if (isset($addedConfig['require-dev'])) {
        $config['require'] = $addedConfig['require-dev'];
        $config = $this->toAllVersions($config);
    }
    return $config;
}

Yep, that makes it pass, but those duplicated if statements are nasty looking. Let’s deal with them.

01
02
03
04
05
06
07
08
09
10
11
12
private function addNewRequires($addedConfig, $config) {
    $config = $this->addRequire($addedConfig, 'require', $config);
    $config = $this->addRequire($addedConfig, 'require-dev', $config);
    return $config;
}
 private function addRequire($addedConfig, $string, $config) {
    if (isset($addedConfig[$string])) {
        $config['require'] = $addedConfig[$string];
        $config = $this->toAllVersions($config);
    }
    return $config;
}

We can be happy again, tests are green and we refactored our code. I think only one test is left to be written. What if we have both “require” and “require-dev” sections in composer.json?

01
02
03
04
05
06
07
08
09
10
11
function testItCanParseComposerJsonWithBothSections() {
    $actual = $this->parseComposerConf(
        '{"require": {
            "Mockery/Mockery": ">=0.7.2"
            },
        "require-dev": {
            "phpunit/phpunit": "3.7.28"
        }}');
    $this->assertEquals('*', json_decode($actual, true)['require']['Mockery/Mockery']);
    $this->assertEquals('*', json_decode($actual, true)['require']['phpunit/phpunit']);
}

That fails because the packages set by “require-dev” will overwrite those of “require” and we will have an error:

1
Undefined index: Mockery/Mockery

Just add a plus sign to merge the arrays, and we are done.

1
2
3
4
5
6
7
private function addRequire($addedConfig, $string, $config) {
    if (isset($addedConfig[$string])) {
        $config['require'] += $addedConfig[$string];
        $config = $this->toAllVersions($config);
    }
    return $config;
}

Tests are passing. Our logic is finished. All we left to do is to extract the methods into their own file and class. The final version of the tests and the SatisUpdater class can be found in the attached source code.

We can now modify our cron script to load our parser and run it on our composer.json. This will be specific to your projects’ particular folders. Here is an example you can adapt to your system.

01
02
03
04
05
06
07
08
09
10
11
#!/usr/local/bin/php
<?php require_once __DIR__ . '/Configuration.php';
$outputDir = '/path/to/your/packages-mirror';
$composerJsonFile = '/path/to/your/projects/composer.json';
$satisConf = '/path/to/your/mirrored-packages.conf';
$satisUpdater = new SatisUpdater();
$conf = $satisUpdater->parseComposerConf(file_get_contents($composerJsonFile)); file_put_contents($satisConf, $conf);
 system(sprintf('/path/to/satis/bin/satis build %s %s', $satisConf, $outputDir), $retval); exit($retval);

Making Your Project Use the Mirror

We talked about a lot of things in this article, but we did not mention how we will instruct our project to use the mirror instead of the Internet. You know, the default is packagist.org? Unless we do something like this:

1
2
3
4
5
6
"repositories": [
     {
         "type": "composer",
         "url": "http://your-mirror-server:4680"
     }
 ],

That will make your mirror the first choise for composer. But adding just that into thecomposer.json of your project will not disable access to packagist.org. If a package can not be found on the local mirror, it will be downloaded from the Internet. If you wish to block this feature, you may also want to add the following line to the repositories section above:

1
"packagist": false

Final Thoughts

That’s it. Local mirror, with automatic adapting and updating packages. Your colleagues will never have to wait a long time while they or the CI server installs all the requirements of your projects. Have fun.

New Development Courses Available on Tuts+ Premium

Currently, more than 15,000 members are sharpening their skills in web design, web development, photography, Photoshop, vector design, video effects and much more. Our dedicated team adds new content weekly, so there’s always something fresh to sink your teeth into. Today, we’re highlighting a few of the latest and greatest course additions to Tuts+ Premium.

New Development Courses

Ember Testing

In this course we’ll learn how to test Ember applications. We’ll review how to set up an Ember app for testing, what testing helpers are available to us, how to write integration and unit testing, and much more.

Machine Learning Distilled

Welcome to “Machine Learning Distilled.” In this course, we’ll review machine learning in a very compressed way. In other words, what is machine learning, why would you want to do it, and how is it done?

Perfect Workflow in Resharper

Are you struggling trying to figure out how to improve your productivity? Do you have a good handle on the C# language, but still feel like it takes you forever to write code. Well, no more. Enter Resharper by Jetbrains. Resharper is a fantastic extension for Visual Studio that will help you to increase your productivity and decrease your development time.

Diving into Octopress

A lot of blogging frameworks are great if you have a large site that needs the features of a content management system, but what if you want something simpler? Let’s talk about Octopress.

Object-Oriented Design in PHP

Object-Oriented Design can offer a developer a structured way to break apart an application into logical, maintainable chunks. In this course, we go over various different OOD techniques, patterns, and concepts to help you improve the sustainability and testability of your code base.

Join Tuts+ Premium for Courses, eBooks, and More

Tuts+ Premium has a huge collection of courses, eBooks, source files and guides on hundreds of creative and technical topics. And we’re adding new content every week.Sign up now and develop your skills today!

Creating a WordPress Theme from Static HTML – Adding Widgets

In this series, you’ve learned how to convert a static HTML file to a WordPress theme and edit the header file.

So far you’ve:

  • prepared your markup for WordPress
  • converted your HTML to PHP and split your file into template files
  • edited the stylesheet and uploaded your theme to WordPress
  • added a loop to your index file
  • added meta tags, the wp_head hook and the site title and description to your header file
  • added a navigation menu.

 

In this tutorial, I’ll show you how to register widget areas in your theme and display them in various locations. You’ll add a widget area to the header for contact details (or whatever you’d like!) and to the sidebar for sidebar widgets.

What You’ll Need

  • your code editor of choice
  • a browser for testing your work
  • a WordPress installation, either local or remote
  • If you’re working locally, you’ll need MAMP, WAMP or LAMP to enable WordPress to run.
  • If you’re working remotely, you’ll need FTP access to your site plus an administrator account in your WordPress installation.

1. Registering Widgets

To register widget areas, you use the register_sidebar() function. You’ll create a function in your functions file to include all of the sidebars you need to register and then you’ll activate this via the widgets_init action hook.

Note: although this function includes the word ‘sidebar’ in its name, that doesn’t mean your widget areas should just be in the sidebar – this is a hangover from the early days of WordPress as a blogging platform when widgets were just in the sidebar.

Open your functions.php file, then add the following code before the line at the very end which reads ?>:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
function wptutsplus_widgets_init() {
    // In header widget area, located to the right hand side of the header, next to the site title and description. Empty by default.
    register_sidebar( array(
        'name' => 'In Header Widget Area',
        'id' => 'in-header-widget-area',
        'description' => 'A widget area located to the right hand side of the header, next to the site title and description.',
        'before_widget' => '<div id="%1$s">',
        'after_widget' => '</div>',
        'before_title' => '<h3>',
        'after_title' => '</h3>',
    ) );
    // Sidebar widget area, located in the sidebar. Empty by default.
    register_sidebar( array(
        'name' => 'Sidebar Widget Area',
        'id' => 'sidebar-widget-area',
        'description' => 'The sidebar widget area',
        'before_widget' => '<div id="%1$s">',
        'after_widget' => '</div>',
        'before_title' => '<h3>',
        'after_title' => '</h3>',
    ) );
}
add_action( 'widgets_init', 'wptutsplus_widgets_init' );
?>

This registers two widget areas: one in the header and one in the sidebar itself. The parameters of register_sidebar() are:

  • name – a unique name for the widget area which will be displayed on the Widgets admin screen.
  • id – a unique ID for the widget area which you’ll use shortly to add your widget area to the correct location in your theme.
  • description – a description to display in the Widgets admin screen.
  • before_widget – the markup to precede each widget in the widget area. This helps with styling.
  • after_widget – the markup to follow each widget area. Make sure you close any elements you opened using the before_widget parameter
  • before_title – markup to precede each widget’s title – I’ve added an h3element with a class which helps with styling.
  • after_title – the markup to close any elements you’ve opened before the widget title.

Now save your functions file.

If you open your site admin, you’ll see that you now have access to the Widgets admin page, which has the two widget areas added:

creating-wordpress-theme-from-static-html-widgets-screen-before

However, any widgets you add here won’t display in your theme yet as you still need to add them to your template files.

2. Adding Widget Areas to Your Template Files

First, I’ll add the widget area for the header and then the sidebar.

Adding a Widget Area to the Header

Open your header.php file and locate the following code:

1
2
3
4
5
<aside class="header widget-area half right" role="complementary">
  <div class="widget-container">
    This will be a widget area in the header - address details (or anything else you like) goes here.
  </div><!-- .widget-container -->
</aside><!-- .half right -->

Replace it with this:

1
2
3
4
5
<?php if ( is_active_sidebar( 'in-header-widget-area' ) ) { ?>
  <aside class="in-header widget-area half right" role="complementary">
    <?php dynamic_sidebar( 'in-header-widget-area' ); ?>
  </aside>
<?php } ?>

Save the file.

Adding the Sidebar Widget Area

Now open your theme’s sidebar.php and find this code:

01
02
03
04
05
06
07
08
09
10
11
</pre>
<aside class="sidebar widget-area one-third right" role="complementary">
  <div class="widget-container">
    <h3 class="widget-title">A Sidebar Widget</h3>
      This is a sidebar widget - in your WordPress theme you can set these up to display across your site.
  </div><!-- .widget-container -->
  <div class="widget-container">
    <h3 class="widget-title">Another Sidebar Widget</h3>
      A second sidebar widget - maybe you could use a plugin to display a social media feed, or simply list your most recent posts.
  </div><!-- .widget-container -->
</aside>

Replace it with the following:

1
2
3
4
5
<?php if ( is_active_sidebar( 'sidebar-widget-area' ) ) { ?>
  <aside class="sidebar widget-area one-third right" role="complementary">
    <?php dynamic_sidebar( 'sidebar-widget-area' ); ?>
  </aside>
<?php } ?>

Finally, save the file.

3. Adding Widgets via the Widgets Admin

The final stage is to add some widgets via the widgets admin screen. I’m going to add two widgets to the header widget area:

  • a text widget with contact details
  • a search box

And I’ll add two widgets to the sidebar:

  • the “Recent Posts” widget
  • the “Meta” widget

Once I’ve done that, my “Widgets” admin screen looks like this:

creating-wordpress-theme-from-static-html-widgets-screen-after

Finally, if I open my site in my browser, I can see the widgets populated in the theme:

creating-wordpress-theme-from-static-html-widgets-in-theme

Some work is still needed on styling – the search bar is a little wide for example, but the content is there. You now have widget areas in your theme!

Summary

In this tutorial, I showed you how to register widget areas and add them to two places in your theme: the header and the sidebar. You could add widget areas anywhere you like in your theme – they can be very useful for adding content across your pages or posts via widgets.

Examples include:

  • a widget area before and after the content, maybe for displaying related posts or social media share buttons or a call to action button
  • a widget area before or after the navigation
  • widget areas in the footer—you’ll add those to this theme in the next tutorial
  • widget areas just for one content type—in Part 11 of this tutorial I’ll show you how to create a custom archive template and you’ll see how different templates for different content types could be combined with multiple widget areas to vary the sidebars in your site.

In the next tutorial, you’ll finish off the footer.php file in your theme, adding multiple widget area in a ‘fat footer’, plus a colophon for copyright information and thewp_footer action hook.

Resources

Android 2013: A Year in Review and 2014 Predictions

It was another strong and, may I say, dominant year for the Android platform. It continued to dominate in market share and climb in numbers. Countless devices were released featuring the popular, mobile operating system and not all of those devices were smartphones or tablets. Android is making headway in new territories. 2013 was a busy year for the green robot. Now that the year has come to a close, it’s time to recap the action.

Market Share

Android’s market share in 2013 was impressive. Google reported that there’d been over 900,000 activations and IDC (International Data Corporation) data showed thatfour out of five phones shipped with the Android operating system on it. There was good news on the Google Play front as well. Google Play is accounting for 75% of all app downloads.

Unfortunately, market share doesn’t equal profit. In 2013, Apple’s App Store generated over $10 billion with December accounting for $1 billion.

From a developer’s perspective, these numbers are encouraging and the same is true for Google Play. However, it remains a fact that Android users spend less money on applications than iOS users. Apple’s mobile platform remains more profitable than Google’s.

API distribution

In 2013, Android made good progress addressing the operating system’s problematic fragmentation. Android versions below 2.2 have virtually disappeared with 2.2 (Froyo) and 3.2 (Honeycomb) following suit.

As of January 2014, more than 50% of Android devices are running a flavor of Jelly Bean, while 2.3 (Gingerbread) and 4.0 (Ice cream sandwich) declined considerable in 2013. This is a good sign, though, as it indicates that attempts to reduce Android’s fragmentation are heading the right direction.

December 2012
Version Codename API Distribution
1.5 Cupcake 3 0.1%
1.6 Donut 4 0.3%
2.1 Eclaire 7 2.7%
2.2 Froyo 8 10.3%
2.2-2.3.2 Ginger Bread 9 0.2%
2.3.3-2.3.7 Ginger Bread 10 50.6%
3.1 Honeycomb 12 0.4%
3.2 Honeycomb 13 1.2%
4.0.3-4.0.4 Ice Cream Sandwich 15 27.5%
4.1 Jelly Bean 16 5.9%
4.2 Jelly Bean 17 0.8%
December 2013
Version Codename API Distribution
2.2 Froyo 8 1.6%
2.3.3-2.3.7 Ginger Bread 10 24.1%
3.2 Honeycomb 13 0.1%
4.0.3-4.0.4 Ice Cream Sandwich 15 18.6%
4.1 Jelly Bean 16 37.4%
4.2 Jelly Bean 17 12.9%
4.3 Jelly Bean 18 4.2%
4.4 KitKat 19 1.1%

Hardware

Nexus Line

The Nexus line was updated with two new entries in 2013. LG’s Nexus 5 was the successor to the Nexus 4 and, once again, it was LG manufacturing the device. The device itself didn’t ship with revolutionary features for its time of launch, it was the KitKat experience and the mid-market price tag of $349 that made for a desirable device. Asus’ Nexus 7 also received a refresh, which included a 1200 x 1920 display and 2GB of RAM.

Gaming

Android made great strides as a gaming platform with a few new and notable gaming systems using Android as their operating system. One of the biggest hits was the Ouya. The tiny console’s goal is to bring cost-effective hardware into the living room and to provide a platform that is open to small developers. The concept proved a major hit onKickstarter. It reached its goal in just eight hours.

And then there was the NVidea Shieldultimate gaming and portable entertainmentas NVidia puts it. The Shield is a powerful handheld gaming console with access to Google play and the ability to stream games from compatible computers. It makes for an impressive and unique device, but the key question is whether it will make a dent in the incredibly competitive gaming landscape.

Glass

Google Glass continued to grow with many developers and technology enthusiasts joining the Glass Explorers Program. Early adopters and developers were excited enough by the product to spend the $1,500 to join the program. Most of the program’s participants continue to give overwhelmingly positive responses so it seems Google is doing something right with Google Glass. People wearing Google Glass still experience social awkwardness and tension and it will therefore be interesting to see if this holds back the adoption and growth of Google Glass.

Galaxy Gear

In 2013, wearables started to become mainstream. Really? No. Not really. However, thePebble smartwatch was finally released after its successful Kickstarter campaign. In addition, Sony released its second generation of smartwatch and Samsung launched the Galaxy Gear in the fall.

None managed to win over the average consumer and they failed to prove the usefulness of a smartwatch to the mass market. This leaves the market wide open for 2014 and with big names like AppleGoogle, and Microsoft rumored to be introducing products in this category, it could become an interesting year for the smartwatch. Let’s hope the competition in this space will lead to spectacular innovation so that we will start to see more useful and practical implementation of smartwatches in this market segment.

Google TV

It was a slow year for Google TV. There were few additions to the market and none that were noteworthy. It was announced that Android version 4.2.2 will eventually make its way to compatible set-top boxes and that it will also get support for Google’s Chromecast at some point.

Chromecast

Speaking of Google’s Chromecast, on July 24, Google announced the Chromecast. The Chromecast is a $35 dongle that allows consumers to stream content from a computer or mobile device to their TV over Wi-Fi. The cheap dongle was very well received and it even made it in TIME’s Top 10 Gadgets of 2013.

KitKat

On October 31, Google released Android 4.4, also known as KitKat. The release came as a surprise to many as Key Lime Pie and 5.0 were expected instead of KitKat. KitKat was also the first version to be named after a branded product.

The update itself is substantial and includes many new features. The update’s theme was reducing the operating system’s footprint and making it more efficient in terms of power consumption and performance on older and low-end devices.

From a developer’s perspective, it is interesting to note the update also introduced new memory management tools, new storage access and printing frameworks, and host card emulation. You can read a more comprehensive review of Android KitKat on Mobiletuts+.

CyanogenMod

One of last year’s most significant success stories came from Android’s community operating system, CyanogenMod. CyanogenMod has been helping Android users for years getting the best from their devices, long after official updates and patches had stopped. In 2013, the people behind the popular Android distribution was able to raise funding and create a company. The not so surprising result is that rumors have started circulating about the possibility of an official CyanogenMod smartphone in the not so distant future.

Android Studio

During last year’s Google I/O developer conference, Google announced a new IDE (Integrated Development Environment) for the Android platform, Android Studio. It was made available to developers in June as an early access preview. The preview is available for download on Windows, OS X, and Linux from the Android developer website.

Android Studio is based on IntelliJ IDEA, a popular Java IDE developed by JetBrains. Based on early reviews and feedback, it seems to be well received by the Android developer community. Despite a plenty of updates, the IDE is still an early preview and prone to bugs. Eclipse is still preferred by most developers.

Google Play Game Services

In 2013, Google Play and a number of other core applications and services received significant updates. These updates took the spotlight at Google I/O 2013. With more than 900,000 daily activations, it was time to refine the user experience and improve the operating system’s services.

The biggest announcement was Google Play game services SDK enabling developers to create games with achievements, leaderboards, real-time multiplayer. Game data can be stored in the cloud and users can use their Google+ account to sign in.

Predictions

Convergence

My prediction is that 2014 will be the year of convergence. There’s currently an Android device for nearly every situation imaginable and the focus for this year will be on making sure that the user experience is less fragmented and more alike across devices.

Devices

We can expect another update of the operating system this year, which will most likely coincide with a refresh of the Nexus line. The Nexus 10 will receive an update or a sibling and the release of a Nexus 6 is a safe bet.

The key players in the smartphone and tablet market will no doubt release new products, but I think it’s the console market that will surprise us this year. There’s a new breed of consoles, microconsoles, that are trying to enter the console market, such as the Nvidia’s ShieldValve’s Steam Machine, and, of course, the crowdfunded Ouya. Valve’s Steam Machine in particular is an interesting concept as it lets manufacturers create consoles based on a detailed specification. Alienware, for example, recently announced that it will be releasing a Steam Machine in September. It will be an interesting year for gamers. That’s for sure.

In 2013, several companies entered the wearables category and this trend will not stop in 2014. The rumors about an Apple smartwatch have subsided a little, but where there’s smoke, there’s fire. Right?

Android

I hope we get to see Android 5.0 in 2014 as this milestone release is long overdue. The update may introduce improvements related to multitasking, better Chromecast integration, and deeper integration of Google Now. As for Google Now, new voice commands will be added as well as improvements to voice search that build upon theOK Google features added to Google Now in 2013.

Many consumers hope to see KitKat on their devices at some point. With the market as competitive as it is, failing to quickly and smoothly update handsets could put several manufactures into trouble. Fast and smooth updates of the operating system remain one of the key advantages iOS has over Android. Long gone are the days when a company could issue a single update to a device in its first year and then leave it at that. Failing to keep handsets updated will definitely cause customers to switch.

Google has invested heavily into making KitKat compatible with a wide range of devices including less powerful low-end devices. It goes without saying that developers support this wholeheartedly. Android fragmentation continues to be one of Android’s primary gripes that also affect consumers. KitKat will hopefully be roled out to most of the new devices that we’ll see in 2014, allowing developers to leverage KitKat’s new features in their applications.

Conclusion

2014 is set to be a frantic year for Android. In many aspects, it will be similar to 2010. This time, however, it won’t be smartphones being released in every shape, size, and form factor. Instead, it’ll be wearables, game consoles, and television sets. We’ll be spoiled with plenty of press releases, lots of excitement, and more than enough early adopters regretting their purchase.

Developers too have a lot to look forward to. The first version of Android Studio will very likely see the light of day in 2014. New devices and possibly new markets will continue to please and tempt developers, but I will be spending 2014 tweaking images for high-resolution displays.

What do you think 2014 will bring? Share your thought in the comment’s below or let me know on Twitter or Google+.