Release notes

Each release of GRR brings some significant changes as the project’s code is moving quickly. This section tries to identify the key changes and issues to watch for when upgrading from one version to another.

Server

3.1.0.2 to 3.2.0.1 (Sep 5 2017)

Starting with 3.2.0.1, we plan on releasing more frequently, since we have automated a large part of the release process. The recommended way of installing this version of GRR is with the server debian package, which includes client templates for all supported platforms. PIP packages, and a Docker image for the release will be made available at a later date.

  • IMPORTANT Before upgrading, make sure to back up your data store. This release has changes that may be incompatible with older versions of the data store. Also make copies of your configs - for reference, and in case you want to roll back the upgrade.
  • WARN Rekall is disabled by default. ‘Rekall.enabled: True’ line should be added to GRR configuration file in order for Rekall to work. There are known stability issues in Rekall and winpmem driver. Disabling it by default makes its ‘in dev/experimental’ status more explicit.
  • WARN Components are deprecated. The GRR client is now monolithic, there’s no need to upload or update components anymore. NOTE: The new GRR server will only be able to run Rekall-based flows (AnalyzeClientMemory, MemoryCollector) with the latest GRR client, since it expects the client not to use components.
  • WARN Rekall flows have been moved to ‘DEBUG’ category. They won’t be visible unless users enable ‘Mode: DEBUG’ in the settings menu (accessible via the settings button in top right corner of GRR AdminUI).
  • WARN A number of backwards-incompatible datastore schema changes were made. The nature of these changes is such that easy data migration between releases is not possible. Data loss of flows and hunts is possible. Please back up your current system if this data is critical.
  • WARN A number of deprecated configuration options were removed. If GRR fails to start after the upgrade, please check the server logs for ‘Unknown configuration option’ messages.
  • WARN Global Flows are no longer supported.
  • INFO Bugfixes, additionals tests and code health work in all parts of the system.
  • INFO HTTP API now covers 100% of GRR functionality. Python API client library significantly extended. API call routers implemented, allowing for flexible authorization.
  • INFO ‘Download as’ functionality added to flows and hunts. Supported formats are CSV, YAML and SQLite.
  • INFO Build process for clients and server is significantly improved. Client templates for all supported platforms, as well as a release-ready server deb, get built on every github commit.

0.3.0-7 to 3.1.0.2

Note that we shifted our version string left to reduce confusion, see the documentation on GRR versions.

  • WARN Quite a few config options have changed. We recommend moving any customizations you have into your server.local.yaml, and using the main config file supplied with the new version. Some important specific ones are called out here.
  • WARN Config option Client.control_urls has been replaced with Client.server_urls. You will need to make sure your polling URL is set in the new variable.
  • WARN We have completely rewritten how memory collection and analysis works inside GRR. Rekall is now responsible for all driver loading, memory collection, and memory analysis. It also replaces the functionality of the Memory Collector flow. This means you must upgrade your clients to be able to do memory collection and analysis.
  • WARN Repacking of old client templates no longer works due to changes necessary for repacking different versions. You will need to use the new client templates that include a build.yaml file.
  • WARN Config option Cron.enabled_system_jobs (a whitelist) was replaced with Cron.disabled_system_jobs (a blacklist) to make adding new cronjobs easier. You will need to remove Cron.enabled_system_jobs and if you made customizations here, translate any jobs you want to remain disabled to the new list.
  • WARN We changed the format for the Knowledge Base protocol buffer to remove a duplication of the users component which was storing users information twice and complicating code. This means that all clients need a fresh interrogate run to populate the machine information. Until the machines run interrogate the %%users.homedir%% and similar expansions won’t work.
  • WARN The compiler we use to build the Mac client template has become more aggressive with optimizations and now emits code optimized to specific processor versions. The mac client will not run on a processor older than our build machine - an early 2011 Macbook with the Intel Sandy Bridge architecture. If you need to run GRR on, for example, Core 2 duo machines, you will need a custom client built on one of those machines.
  • WARN We no longer support running the client on Windows XP, it’s difficult to make it work and we have no use for an XP client ourselves. See here for our OS support statement.
  • INFO Strict context checking was added for config files in this commit, which exposed a number of minor config typo bugs. If you get InvalidContextError on upgrade, you need to update your config to fix up the typos. The config/contexts.py file is the canonical reference.

Upgrade procedure:

  1. Make backups of important data such as your configs and your database.
  2. Upgrade to xenial
  3. Clear out any old installations: sudo rm -rf /usr/local/bin/grr_*; sudo rm -rf /usr/local/lib/python2.7/dist-packages/grr/
  4. Install deb package
  5. Backup then remove your /etc/grr/grr-server.yaml. Make sure any customizations you made are stored in /etc/grr/server.local.yaml.
  6. Run grr_config_updater initialize

0.3.0-6 to 0.3.0-7

  • INFO We moved to using the open source binplist, rather than bundling it with GRR. If you are getting “ImportError: cannot import name binplist”, make sure you have installed the binplist package and deleted grr/parsers/binplist* to clean up any .pyc files that might have been created.
  • INFO If you have troubles with the rekall efilter library failing to import, try deleting all of your rekall-related .pyc’s and make sure you’re installing the version in requirements.txt. See this bug for full details.

0.3.0-5 to 0.3.0-6

This version bump was to keep in step with the client, which got a new version because of a memory collection bug.

  • WARN The artifact format changed in a way that is not backwards compatible. You’ll need to delete any artifacts you have in the artifact manager, either through the GUI or on the console with aff4.FACTORY.Delete("aff4:/artifact_store") before you upgrade. If you forget you should be able to use the console or you can roll back to this commit before the artifact change.
  • WARN Keyword search is now based on an index that is built at weekly interrogate time, which means decomissioned clients won’t appear in search results. To populate all of your historical clients into the search index, use the code snippet on this commit.

0.3.0 to 0.3.0-5

  • WARN Rekall made some fundamental changes that mean rekall in the 0.3.0-5 server won’t work with rekall in 3.0.0.3 clients, so a client upgrade to 3.0.0.5 is required. Non-rekall GRR components should continue to work.
  • WARN The enroller component was removed, and is now handled by the worker. You will need to stop those jobs and delete the relevant upstart/init scripts.
  • INFO We now check that all config options are defined in the code and the server won’t start if they aren’t. When you upgrade it’s likely there is some old config that will need cleaning up. See the graveyard below for advice.

0.2.9 to 0.3.0

  • WARN After install of new deb you will need to restart the service manually.
  • INFO To get the new clients you will want to repack the new version with sudo grr_config_updater repack_clients. This should give you new Windows, Linux and OSX clients.
  • INFO Client themselves will not automatically be upgraded but should continue to work.

0.2-8 to 0.2-9

  • WARN After install of new deb you will need to restart the service manually.
  • WARN Users have changed from being managed in the config file to being managed in the datastore. This means your users will not work. We haven’t migrated them automatically, please re-add with sudo grr_config_updater add_user <user>
  • INFO To get the new clients you will want to repack the new version with sudo grr_config_updater repack_clients. This should give you new Windows, Linux and OSX clients.
  • INFO Client themselves will not automatically be upgraded but should continue to work.

Client

3.2.0.1 -

Starting from 3.2.0.1, we the same version string for the client and server, since they get released at the same time.

3.0.7.1 to 3.1.0.0

Note that we skipped some numbers to make versioning simpler and reduce confusion, see the documentation on GRR versions.

  • WARN We changed rekall to be a independently updatable component in the client, which is a backwards incompatible change. You must upgrade your clients to 3.1.0.0 if you want to use memory capabilities in the 3.1.0 server.
  • WARN Our previous debian package added the GRR service using both upstart and init.d runlevels. This caused some confusion on systems with ubuntu upstart systems. We detect and remove this problem automatically with the new version, but since it is a config file change you need to specify whether to install the new config or keep the old one, or you will get a config change prompt. New is preferred. Something like sudo apt-get -o Dpkg::Options::="--force-confnew" grr.

3.0.0.7 to 3.0.7.1

  • INFO 3.0.7.1 is the first version of GRR that will work on OS X El Capitan. The new OS X System Integrity Protection meant we had to shift our install location from /usr/lib/ to /usr/local/lib.
  • INFO We changed our version numbering scheme for the client at this point to give us the ability to indicate multiple client versions that work with a server version. So 3.0.7.* clients will all work with server 0.3.0-7.

3.0.0.6 to 3.0.0.7

  • WARN Linux and OS X clients prior to 3.0.0.7 were using /etc/grr/client.local.yaml as the local writeback location. For 3.0.0.7 this was changed to /etc/%(Client.name).local.yaml where the default is /etc/grr.local.yaml. If you wish to preserve the same client IDs you need to use platform management tools to copy the old config into the new location for all clients before you upgrade. If you don’t do this the clients will just re-enrol and get new client IDs automatically.

3.0.0.5 to 3.0.0.6

  • INFO 3.0.0.5 had a bug that broke memory collection, fixed in this commit. We also wrote a temporary server-side workaround, so upgrading isn’t mandatory. 3.0.0.5 clients should still work fine.

3.0.0.3 to 3.0.0.5

(We skipped a version number, there’s no 3.0.0.4)

3.0.0.2 to 3.0.0.3

  • WARN A change to OpenSSL required us to sign our CSRs generated during the enrollment process. This wasn’t necessary previously and provided no benefit for GRR so we had gained some speed by not doing it. Since new OpenSSL required it, we started signing the CSRs, but it meant that the 3.0.0.3 server will reject any 3.0.0.2 clients that haven’t already enrolled (i.e. they will see a HTTP 406). Old 3.0.0.2 clients that have already enrolled and new 3.0.0.3 clients will work fine. This basically just means that you need to push out new clients at the same time as you upgrade the server.

Config Variable Graveyard

Sometimes config variables get renamed, sometimes removed. When this happens we’ll try to record it here, so users know if local settings should be migrated/ignored etc.

You can verify your config with this (root is required to read the writeback config)

sudo PYTHONPATH=. python ./run_tests.py --test=BuildConfigTests.testAllConfigs
  • AdminUI.team_name: replaced by Email.signature
  • ClientBuilder.build_src_dir: unused, effectively duplicated ClientBuilder.source
  • ClientBuilder.executables_path: ClientBuilder.executables_dir
  • Client.config: unused. Now built from Client.config_hive and Client.config_key
  • Client.config_file_name: unused
  • Client.location: replaced by Client.control_urls
  • Client.package_maker_organization: replaced by ClientBuilder.package_maker_organization
  • Client.tempdir: replaced by Client.grr_tempdir and Client.tempdir_roots
  • Email.default_domain: essentially duplicated Logging.domain, use that instead.
  • Frontend.processes: unused
  • Nanny.nanny_binary: replaced by Nanny.binary
  • NannyWindows.* : replaced by Nanny.
  • PyInstaller.build_root_dir: unused, effectively duplicated ClientBuilder.build_root_dir.
  • Users.authentication: unused, user auth is now based on aff4:/users objects. Use config_updater to modify them.
  • Worker.task_limit: unused
  • Worker.worker_process_count: unused
  • Cron.enabled_system_jobs (a whitelist) was replaced with Cron.disabled_system_jobs (a blacklist). Cron.enabled_system_jobs should be removed. Any custom jobs you want to stay disabled should be added to Cron.enabled_system_jobs.
  • Client.control_urls: renamed to Client.server_urls.