New CKAN open data portal for Queensland Government

The players

Queensland’s DHPW looks after a broad range of areas, including digital technology and services. This incorporates Queensland’s open data policy and standards, and the promotion of open data and shared datasets through Queensland’s open data portal.

The problem

DHPW wanted improvements to the data portal to make it easier to manage and easier to use, to drive open data uptake in Queensland. DHPW had identified specific functionality that would help them manage the platform and interact with users. These requirements were:

• An updated CKAN instance

• Integration with the Open Data Institute to display open data certificate badges to users (read our blog on open data certificates for more information)

• Installation and customisation of numerous extensions (including comments, structured data, data requests and data validation)

• Improved, customised Google Analytics for the portal

The solution

Following our initial discovery, the main project build was split into six build sprints to deliver on the functional requirements. Each sprint focused on delivering one or two requirements to provide improved features and a better user experience. The main deliverables were:

• Fresh install of CKAN, including customisations: Early in the project the decision was made to use the latest stable CKAN release (2.8.3 at time of writing) and start with a clean slate, excluding all existing CKAN customisations and porting over as necessary.

• Implementation of custom CKAN theme:The updated data portal includes a new design that’s consistent across the data portal and content management system. A custom CKAN theme was implemented to provide the user with a seamless experience between the two sites.

• Installation and configuration of the structured data extension: Modification of the dataset and resource schemas.

• Integration of the open data certificate API to help users verify the quality of the dataset: A custom CKAN extension was created (and subsequently contributed back to the CKAN community) to easily display ODI certificate badges on datasets if available.

• Installation and customisation of the data requests extension: Data requests enable site users to request datasets that may not already exist on the data portal. Data.qld uses the standard CKAN data request extension to facilitate this, along with some enhancements including automatic assignment to an internal organisation (for further allocation), the ability to re-open and re-assign data requests, and additional email notifications to support these workflows.

• Installation and extension of comments extension for datasets and data requests: The “Disqus” extension seems to be the go to for enabling commenting within CKAN sites, however Data.qld wanted to retain and control the comment data internally. Salsa revived an existing, incomplete CKAN comments extension and extended it to be more secure, along with adding extra functionality such as profanity filtering, email notifications, inappropriate comment flagging, etc.

• Installation and customisation of data validation extension: Valid data is an important consideration when opening up datasets to the public as it can increase end-user confidence in the quality of the data. Salsa used the CKAN validation extension along with some enhancements to improve the user experience for both data custodians and end users.

• Google Analytics optimisations: Data.qld uses Google Tag Manager (GTM) to track analytical data regarding site usage. Salsa included the GTM tracking code and implemented a custom solution to retain tracking of important API calls within Google Analytics.

• Implementation of an automated test framework to reduce overall regression and release costs.

Importantly, the updated portal is a more ‘vanilla’ CKAN instance focused on out-of-the-box features and customising those (see our CKAN viability blog), rather than custom-built functionality that drives up costs and maintenance requirements in the long-run (see our blog on best practice for custom requirements for more information).

Salsa delivered all functionality defined in the project scope well under budget, which allowed for the delivery of additional improvements for the data portal.

Giving back — open source ethos

During the project, we also identified gaps in existing modules and improved these modules as part of our commitment to the open source movement and giving back to open source. This is also in line with our 20% giveback pledge.

Some examples of our open source contributions are:

1. The `ckanext-odi-certificates` extension. This was developed by Salsa Digital to allow CKAN sites to display an Open Data Institute (ODI) certificate badge on their datasets if the dataset has a corresponding certificate in the ODI certificate repository (https://certificates.theodi.org/en/). This extension has been cleaned up and made generic to be provided back to the CKAN community for use on other CKAN sites.

2. The workflow customisations introduced to the data requests extension have been isolated into features and submitted back to the original extension developers as pull requests to be incorporated into future releases of the extension.

3. The enhancements added to the comments extension have been isolated into features and submitted back to the original extension developers as pull requests to be incorporated into future releases of the extension.

4. Other minor display changes and tweaks have been isolated into Github Public Gists to contribute that knowledge back to the CKAN community through blog posts, answers to Stack Overflow questions, responses to CKAN developer mailing list questions, etc.

The benefits

The open data portal updates provide the following benefits to open data users, DHPW and to the Queensland Government:

• New CKAN instance built on CKAN2.8, which provides increased security and can be easily upgraded in the future

• Updated portal that’s easier to maintain

• Improved platform stability through an automated test framework (when changes are made any regression issues are easily identified)

• More interactive experience for users, who can now request data and use the new comments feature to interact with the DHPW data team

• More tailored data experience, with the DHPW data team using the comment feature to ensure data is relevant for users

• Data quality verification through the integration of open data certificate API

• More insights into what’s happening on the site through the optimised Google Analytics

Why Salsa Digital?

Salsa won this project based on our tender response, which demonstrated our CKAN expertise and track record in the government space. Salsa is also known for our commitment to open source, open data and open government and this was a key contributing factor. In fact, as part of this project Salsa pledged to help raise awareness of Queensland’s open data through an open data blog series. This was the start of our Open Data Insights series, which will profile datasets from around Australia, including Queensland. Our first Open Data Insight looked at Queensland crash statistics.

About Salsa Digital

We’re an open source digital agency focused on helping governments and enterprises become more open, more connected and more consolidated. We’re committed to the open government movement and to the many benefits this ethos provides including transparency, innovation, and sharing and improving problems and patterns solved by the pioneers before us.

The open source movement has played a key role in the evolution of Salsa. Over the years, as technology and practices have developed, this commitment to open source has grown to encompass a much broader base beyond open source CMSs — open data, open platform and open design. We contribute and deliver services in all these areas, with a strong involvement in specific open source initiatives such as Drupal, Wordpress, GovCMS, Single Digital Presence, Kubernetes/Lagoon and CKAN.